Jul 06 15:26:18.262832 ip-10-0-129-225 systemd[1]: Starting Kubernetes Kubelet... Jul 06 15:26:18.611753 ip-10-0-129-225 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 15:26:18.611753 ip-10-0-129-225 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 15:26:18.611753 ip-10-0-129-225 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 15:26:18.611753 ip-10-0-129-225 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 15:26:18.611753 ip-10-0-129-225 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 15:26:18.612605 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.612517 2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 15:26:18.618906 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618891 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618908 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618912 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618915 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618918 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618921 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618924 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618927 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618930 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618934 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618938 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618941 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618943 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618946 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618949 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618951 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618955 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618958 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:26:18.618950 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618960 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618963 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618967 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618970 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618975 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618978 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618981 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618984 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618988 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618991 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618994 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.618997 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619000 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619003 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619006 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619008 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619010 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619013 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619015 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:26:18.619427 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619018 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619021 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619023 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619026 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619028 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619031 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619033 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619036 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619038 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619041 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619044 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619046 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619048 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619051 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619053 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619056 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619059 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619061 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619064 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619066 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:26:18.619962 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619069 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619071 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619073 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619076 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619078 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619081 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619083 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619085 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619088 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619090 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619092 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619095 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619097 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619099 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619102 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619105 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619107 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619118 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619120 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619123 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:26:18.620494 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619125 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619128 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619130 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619133 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619135 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619138 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619140 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619143 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619146 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619148 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619151 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619153 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619155 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619159 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619161 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619163 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619166 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619168 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619171 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619173 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:26:18.621023 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619175 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.619178 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620369 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620376 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620380 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620383 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620386 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620389 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620392 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620395 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620398 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620401 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620403 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620406 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620409 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620411 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620413 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620416 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620418 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620422 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:26:18.621565 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620425 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620428 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620430 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620432 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620435 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620437 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620440 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620442 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620446 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620449 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620452 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620455 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620458 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620460 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620463 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620466 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620468 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620470 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620473 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620475 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:26:18.622136 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620478 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620481 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620483 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620485 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620488 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620490 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620493 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620495 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620498 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620500 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620504 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620507 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620509 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620512 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620514 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620517 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620519 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620522 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620540 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:26:18.622713 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620543 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620547 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620550 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620553 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620556 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620558 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620561 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620564 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620566 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620569 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620571 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620574 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620576 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620579 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620582 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620584 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620586 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620588 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620591 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620594 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:26:18.623216 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620596 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620598 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620602 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620604 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620607 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620610 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620614 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620617 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620620 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620622 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620625 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620627 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620630 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620632 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620635 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620637 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620640 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620642 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620645 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:26:18.623768 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620647 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620649 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.620652 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621011 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621015 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621017 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621020 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621023 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621026 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621031 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621034 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621037 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621040 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621043 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621046 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621048 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621050 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621055 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621058 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:26:18.624260 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621061 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621064 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621066 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621069 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621071 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621074 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621076 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621079 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621081 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621084 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621086 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621088 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621091 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621093 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621096 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621098 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621101 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621103 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621105 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621108 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:26:18.624777 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621111 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621114 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621116 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621118 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621121 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621124 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621126 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621129 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621132 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621135 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621139 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621141 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621144 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621146 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621149 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621151 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621154 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621157 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621161 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:26:18.625290 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621164 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621167 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621170 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621172 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621175 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621177 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621180 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621182 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621184 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621187 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621189 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621192 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621194 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621198 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621200 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621203 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621205 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621208 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621210 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621213 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:26:18.625788 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621215 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621217 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621220 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621224 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621226 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621228 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621231 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621233 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621235 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621238 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621240 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621243 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621246 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621248 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621250 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621253 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621255 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621258 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621260 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621262 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:26:18.626315 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621265 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621267 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621270 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621272 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621338 2561 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621346 2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621352 2561 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621356 2561 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621360 2561 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621363 2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621367 2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621371 2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621374 2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621376 2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621380 2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621384 2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621387 2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621390 2561 flags.go:64] FLAG: --cgroup-root="" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621393 2561 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621395 2561 flags.go:64] FLAG: --client-ca-file="" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621398 2561 flags.go:64] FLAG: --cloud-provider="external" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621401 2561 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621405 2561 flags.go:64] FLAG: --cluster-domain="" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621408 2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 15:26:18.626966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621411 2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621414 2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621417 2561 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621420 2561 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621423 2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621426 2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621429 2561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621432 2561 flags.go:64] FLAG: --contention-profiling="false" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621435 2561 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621448 2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621452 2561 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621454 2561 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621458 2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621462 2561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621465 2561 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621468 2561 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621471 2561 flags.go:64] FLAG: --enable-server="true" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621474 2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621478 2561 flags.go:64] FLAG: --event-burst="100" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621481 2561 flags.go:64] FLAG: --event-qps="50" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621483 2561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621486 2561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621489 2561 flags.go:64] FLAG: --eviction-hard="" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621493 2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 15:26:18.627616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621496 2561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621499 2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621502 2561 flags.go:64] FLAG: --eviction-soft="" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621505 2561 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621507 2561 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621510 2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621513 2561 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621516 2561 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621519 2561 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621521 2561 flags.go:64] FLAG: --feature-gates="" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621539 2561 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621543 2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621545 2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621548 2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621551 2561 flags.go:64] FLAG: --healthz-port="10248" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621554 2561 flags.go:64] FLAG: --help="false" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621557 2561 flags.go:64] FLAG: --hostname-override="ip-10-0-129-225.ec2.internal" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621560 2561 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621562 2561 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621565 2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621569 2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621573 2561 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621576 2561 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 15:26:18.628233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621579 2561 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621582 2561 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621584 2561 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621587 2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621590 2561 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621593 2561 flags.go:64] FLAG: --kube-reserved="" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621596 2561 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621598 2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621602 2561 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621605 2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621608 2561 flags.go:64] FLAG: --lock-file="" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621610 2561 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621613 2561 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621616 2561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621620 2561 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621623 2561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621626 2561 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621629 2561 flags.go:64] FLAG: --logging-format="text" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621631 2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621635 2561 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621637 2561 flags.go:64] FLAG: --manifest-url="" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621640 2561 flags.go:64] FLAG: --manifest-url-header="" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621644 2561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621647 2561 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621650 2561 flags.go:64] FLAG: --max-pods="110" Jul 06 15:26:18.628898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621653 2561 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621656 2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621658 2561 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621661 2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621664 2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621667 2561 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621670 2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621677 2561 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621680 2561 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621683 2561 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621685 2561 flags.go:64] FLAG: --pod-cidr="" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621688 2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621693 2561 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621696 2561 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621698 2561 flags.go:64] FLAG: --pods-per-core="0" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621701 2561 flags.go:64] FLAG: --port="10250" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621704 2561 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621707 2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05689aa0811f828bd" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621710 2561 flags.go:64] FLAG: --qos-reserved="" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621713 2561 flags.go:64] FLAG: --read-only-port="10255" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621716 2561 flags.go:64] FLAG: --register-node="true" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621718 2561 flags.go:64] FLAG: --register-with-taints="" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621721 2561 flags.go:64] FLAG: --registry-burst="10" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621724 2561 flags.go:64] FLAG: --registry-qps="5" Jul 06 15:26:18.629559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621727 2561 flags.go:64] FLAG: --reserved-cpus="" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621729 2561 flags.go:64] FLAG: --reserved-memory="" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621732 2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621735 2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621738 2561 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621741 2561 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621743 2561 flags.go:64] FLAG: --runonce="false" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621746 2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621749 2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621751 2561 flags.go:64] FLAG: --seccomp-default="false" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621754 2561 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621756 2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621759 2561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621762 2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621765 2561 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621768 2561 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621771 2561 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621773 2561 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621776 2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621779 2561 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621782 2561 flags.go:64] FLAG: --system-cgroups="" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621784 2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621789 2561 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621792 2561 flags.go:64] FLAG: --tls-cert-file="" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621794 2561 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 15:26:18.630191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621799 2561 flags.go:64] FLAG: --tls-min-version="" Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621802 2561 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621804 2561 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621807 2561 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621809 2561 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621812 2561 flags.go:64] FLAG: --v="2" Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621815 2561 flags.go:64] FLAG: --version="false" Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621819 2561 flags.go:64] FLAG: --vmodule="" Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621822 2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.621825 2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621930 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621935 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621939 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621942 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621945 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621948 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621951 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621953 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621955 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621958 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621960 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:26:18.630843 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621963 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621965 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621968 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621970 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621973 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621975 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621977 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621980 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621983 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621985 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621988 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621990 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621992 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621995 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.621997 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622000 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622002 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622005 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622007 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622010 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:26:18.631405 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622012 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622015 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622017 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622019 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622022 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622024 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622027 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622029 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622031 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622034 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622037 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622041 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622044 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622046 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622049 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622052 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622054 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622057 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622059 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622061 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:26:18.632138 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622064 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622068 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622070 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622073 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622075 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622077 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622080 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622082 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622084 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622087 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622089 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622091 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622094 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622096 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622098 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622101 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622103 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622106 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622108 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622110 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:26:18.632770 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622113 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622115 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622118 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622120 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622124 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622126 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622129 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622131 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622133 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622136 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622138 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622141 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622144 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622148 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622151 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622153 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622156 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622158 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622161 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:26:18.633291 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622163 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:26:18.633820 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622165 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:26:18.633820 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622168 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:26:18.633820 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622170 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:26:18.633820 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622173 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:26:18.633820 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622175 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:26:18.633820 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622177 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:26:18.633820 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622180 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:26:18.633820 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.622184 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:26:18.633820 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.622717 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 15:26:18.633820 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.629115 2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 15:26:18.633820 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.629128 2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 15:26:18.633820 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.629141 2561 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 15:26:18.633820 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.629147 2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 15:26:18.633820 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629200 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629205 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629208 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629211 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629214 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629217 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629220 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629222 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629225 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629228 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629230 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629234 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629236 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629238 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629241 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629243 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629246 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629248 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629250 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:26:18.634193 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629253 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629255 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629258 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629260 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629262 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629265 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629267 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629270 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629273 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629275 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629279 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629281 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629284 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629286 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629288 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629291 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629293 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629295 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629298 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629300 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:26:18.634716 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629303 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629305 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629309 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629313 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629316 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629319 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629321 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629324 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629327 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629330 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629333 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629335 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629338 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629340 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629343 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629345 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629347 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629350 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629352 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:26:18.635251 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629354 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629357 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629359 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629362 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629366 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629369 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629371 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629373 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629376 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629378 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629381 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629383 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629386 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629388 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629390 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629393 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629395 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629397 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629401 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629404 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:26:18.635778 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629407 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629409 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629412 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629414 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629417 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629419 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629421 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629424 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629427 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629429 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629431 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629434 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629437 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629439 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629442 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629445 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629447 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629450 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629453 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:26:18.636335 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629456 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:26:18.636846 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629459 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:26:18.636846 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.629464 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 15:26:18.636846 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629592 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:26:18.636846 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629597 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:26:18.636846 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629600 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:26:18.636846 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629603 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:26:18.636846 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629605 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:26:18.636846 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629608 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:26:18.636846 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629610 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:26:18.636846 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629612 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:26:18.636846 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629614 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:26:18.636846 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629617 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:26:18.636846 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629621 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:26:18.636846 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629624 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629627 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629630 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629633 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629635 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629638 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629641 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629644 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629647 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629649 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629652 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629654 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629656 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629659 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629661 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629664 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629666 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629669 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629671 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629673 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:26:18.637217 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629677 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629680 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629682 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629684 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629687 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629689 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629691 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629693 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629696 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629698 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629701 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629703 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629705 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629707 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629709 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629712 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629714 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629717 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629719 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629723 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:26:18.637894 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629726 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629729 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629732 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629734 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629737 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629739 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629742 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629745 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629748 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629750 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629753 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629755 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629758 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629760 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629763 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629765 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629768 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629770 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629772 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629775 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:26:18.638404 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629777 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629779 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629782 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629784 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629786 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629788 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629791 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629793 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629796 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629798 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629800 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629803 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629805 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629807 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629810 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629812 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629814 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629817 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629819 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629822 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:26:18.638934 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629824 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:26:18.639454 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629826 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:26:18.639454 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629829 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:26:18.639454 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629832 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:26:18.639454 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629834 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:26:18.639454 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629836 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:26:18.639454 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629838 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:26:18.639454 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:18.629841 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:26:18.639454 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.629845 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 15:26:18.639454 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.629958 2561 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 15:26:18.639454 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.633140 2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 15:26:18.639454 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.634542 2561 server.go:1019] "Starting client certificate rotation" Jul 06 15:26:18.639454 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.634634 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 15:26:18.639454 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.634667 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 15:26:18.639840 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.636965 2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 15:26:18.639840 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.639522 2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 15:26:18.647848 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.647832 2561 log.go:25] "Validated CRI v1 runtime API" Jul 06 15:26:18.649905 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.649890 2561 log.go:25] "Validated CRI v1 image API" Jul 06 15:26:18.653480 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.653462 2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 15:26:18.655493 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.655473 2561 fs.go:135] Filesystem UUIDs: map[4a0c7c40-4161-4c0c-8741-e8129f8e7b16:/dev/nvme0n1p3 72fbdbb5-5dd2-4c8a-81be-1fab89e037db:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 06 15:26:18.655585 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.655492 2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 15:26:18.661475 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.661363 2561 manager.go:216] Machine: {Timestamp:2026-07-06 15:26:18.660198931 +0000 UTC m=+0.292750906 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100222 MemoryCapacity:32812183552 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e5b5c398757b1538b8b7914321e58 SystemUUID:ec2e5b5c-3987-57b1-538b-8b7914321e58 BootID:375c8775-2729-479a-b67e-6eaf4a7e600e Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406093824 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f0:8b:43:59:f3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f0:8b:43:59:f3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:87:cb:64:75:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812183552 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 15:26:18.661475 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.661469 2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 15:26:18.661661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.661583 2561 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 15:26:18.662632 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.662607 2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 15:26:18.662769 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.662634 2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-225.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 15:26:18.663357 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.663346 2561 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 15:26:18.663392 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.663359 2561 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 15:26:18.663392 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.663366 2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 15:26:18.663623 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.663606 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 15:26:18.663971 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.663961 2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 15:26:18.665407 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.665397 2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 15:26:18.669379 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.669368 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 06 15:26:18.669496 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.669487 2561 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 15:26:18.669556 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.669514 2561 kubelet.go:486] "Attempting to sync node with API server" Jul 06 15:26:18.669556 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.669547 2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 15:26:18.669661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.669571 2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 15:26:18.669661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.669595 2561 kubelet.go:398] "Adding apiserver pod source" Jul 06 15:26:18.669661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.669616 2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 15:26:18.669990 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.669975 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 15:26:18.670048 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.670037 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 15:26:18.673283 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.673245 2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 15:26:18.675709 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.675694 2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 15:26:18.675782 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.675724 2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 15:26:18.677486 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.677473 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 15:26:18.677578 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.677489 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 15:26:18.677578 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.677495 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 15:26:18.677578 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.677500 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 15:26:18.677578 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.677508 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 15:26:18.677578 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.677515 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 15:26:18.677578 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.677543 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 15:26:18.677578 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.677549 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 15:26:18.677578 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.677556 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 15:26:18.677578 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.677561 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 15:26:18.677578 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.677569 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 15:26:18.677578 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.677574 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 15:26:18.678220 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.678208 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 15:26:18.678220 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.678219 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 15:26:18.679630 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.679605 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 15:26:18.679718 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.679702 2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-225.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 15:26:18.679764 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.679702 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-225.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 15:26:18.681356 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.681343 2561 server.go:1268] "Started kubelet" Jul 06 15:26:18.681426 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.681403 2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 15:26:18.681470 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.681425 2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 15:26:18.681500 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.681488 2561 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 15:26:18.682240 ip-10-0-129-225 systemd[1]: Started Kubernetes Kubelet. Jul 06 15:26:18.683282 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.683265 2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 15:26:18.684024 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.684010 2561 server.go:310] "Adding debug handlers to kubelet server" Jul 06 15:26:18.687002 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.686977 2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 15:26:18.691454 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.691435 2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 15:26:18.691454 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.691447 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 15:26:18.691651 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.691557 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-225.ec2.internal\" not found" Jul 06 15:26:18.692152 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.692133 2561 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 15:26:18.692152 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.692155 2561 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 15:26:18.692276 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.692158 2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 15:26:18.692276 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.692261 2561 reconstruct.go:97] "Volume reconstruction finished" Jul 06 15:26:18.692276 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.692269 2561 reconciler.go:29] "Reconciler: start to sync state" Jul 06 15:26:18.692605 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.692581 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tzcx6" Jul 06 15:26:18.693951 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.693937 2561 factory.go:153] Registering CRI-O factory Jul 06 15:26:18.693951 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.693953 2561 factory.go:223] Registration of the crio container factory successfully Jul 06 15:26:18.694065 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.694002 2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 15:26:18.694065 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.694013 2561 factory.go:55] Registering systemd factory Jul 06 15:26:18.694065 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.694020 2561 factory.go:223] Registration of the systemd container factory successfully Jul 06 15:26:18.694065 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.694035 2561 factory.go:103] Registering Raw factory Jul 06 15:26:18.694065 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.694045 2561 manager.go:1196] Started watching for new ooms in manager Jul 06 15:26:18.694710 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.694697 2561 manager.go:349] Starting recovery of all containers Jul 06 15:26:18.694843 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.694823 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 15:26:18.694892 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.694871 2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-225.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 15:26:18.695795 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.694908 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-225.ec2.internal.18bfbccfecee83e6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-225.ec2.internal,UID:ip-10-0-129-225.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-225.ec2.internal,},FirstTimestamp:2026-07-06 15:26:18.68131223 +0000 UTC m=+0.313864203,LastTimestamp:2026-07-06 15:26:18.68131223 +0000 UTC m=+0.313864203,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-225.ec2.internal,}" Jul 06 15:26:18.700903 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.700877 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tzcx6" Jul 06 15:26:18.706179 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.706026 2561 manager.go:354] Recovery completed Jul 06 15:26:18.710375 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.710364 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:26:18.712154 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.712140 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:26:18.712215 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.712166 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:26:18.712215 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.712176 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:26:18.712698 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.712670 2561 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 15:26:18.712698 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.712679 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:26:18.712698 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.712684 2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 15:26:18.712698 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.712699 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 06 15:26:18.713252 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.713236 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:26:18.713320 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.713261 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:26:18.713320 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.713270 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:26:18.714367 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.714312 2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-225.ec2.internal.18bfbccfeec51f0b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-225.ec2.internal,UID:ip-10-0-129-225.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-225.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-225.ec2.internal,},FirstTimestamp:2026-07-06 15:26:18.712153867 +0000 UTC m=+0.344705841,LastTimestamp:2026-07-06 15:26:18.712153867 +0000 UTC m=+0.344705841,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-225.ec2.internal,}" Jul 06 15:26:18.714718 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.714706 2561 policy_none.go:49] "None policy: Start" Jul 06 15:26:18.714781 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.714725 2561 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 15:26:18.714781 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.714738 2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 15:26:18.716735 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.716718 2561 policy_none.go:47] "Start" Jul 06 15:26:18.756128 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.756114 2561 manager.go:337] "Starting Device Plugin manager" Jul 06 15:26:18.756197 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.756151 2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 15:26:18.756197 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.756160 2561 server.go:87] "Starting device plugin registration server" Jul 06 15:26:18.756376 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.756357 2561 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 15:26:18.756485 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.756373 2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 15:26:18.756485 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.756457 2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 15:26:18.756600 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.756559 2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 15:26:18.756600 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.756568 2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 15:26:18.756997 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.756977 2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 15:26:18.757090 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.757021 2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-225.ec2.internal\" not found" Jul 06 15:26:18.791820 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.791783 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 15:26:18.800567 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.793086 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 15:26:18.800567 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.793128 2561 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 15:26:18.800567 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.793155 2561 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 15:26:18.800567 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.793193 2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 15:26:18.800567 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.795283 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:26:18.857345 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.857326 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:26:18.858051 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.858036 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:26:18.858122 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.858061 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:26:18.858122 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.858071 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:26:18.858122 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.858090 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-225.ec2.internal" Jul 06 15:26:18.865938 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.865902 2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-225.ec2.internal" Jul 06 15:26:18.865938 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.865927 2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-225.ec2.internal\": node \"ip-10-0-129-225.ec2.internal\" not found" Jul 06 15:26:18.876595 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.876574 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-225.ec2.internal\" not found" Jul 06 15:26:18.893378 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.893357 2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-225.ec2.internal"] Jul 06 15:26:18.893426 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.893409 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:26:18.894127 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.894113 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:26:18.894202 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.894136 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:26:18.894202 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.894145 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:26:18.896502 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.896490 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:26:18.896645 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.896631 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal" Jul 06 15:26:18.896682 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.896658 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:26:18.897267 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.897252 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:26:18.897324 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.897277 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:26:18.897324 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.897285 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:26:18.897382 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.897253 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:26:18.897382 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.897354 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:26:18.897382 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.897366 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:26:18.899444 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.899427 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-225.ec2.internal" Jul 06 15:26:18.899550 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.899454 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:26:18.900870 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.900855 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:26:18.900957 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.900883 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:26:18.900957 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.900896 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:26:18.921039 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.921024 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-225.ec2.internal\" not found" node="ip-10-0-129-225.ec2.internal" Jul 06 15:26:18.925286 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.925270 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-225.ec2.internal\" not found" node="ip-10-0-129-225.ec2.internal" Jul 06 15:26:18.977600 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:18.977582 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-225.ec2.internal\" not found" Jul 06 15:26:18.993499 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.993484 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b5b4cb1adc7cdefada5f0eeb4f96fb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal\" (UID: \"7b5b4cb1adc7cdefada5f0eeb4f96fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal" Jul 06 15:26:18.993572 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.993508 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5b4cb1adc7cdefada5f0eeb4f96fb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal\" (UID: \"7b5b4cb1adc7cdefada5f0eeb4f96fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal" Jul 06 15:26:18.993572 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:18.993546 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91375dee49182778f247ddc7d3913f13-config\") pod \"kube-apiserver-proxy-ip-10-0-129-225.ec2.internal\" (UID: \"91375dee49182778f247ddc7d3913f13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-225.ec2.internal" Jul 06 15:26:19.078381 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:19.078352 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-225.ec2.internal\" not found" Jul 06 15:26:19.093664 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.093643 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b5b4cb1adc7cdefada5f0eeb4f96fb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal\" (UID: \"7b5b4cb1adc7cdefada5f0eeb4f96fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal" Jul 06 15:26:19.093710 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.093682 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b5b4cb1adc7cdefada5f0eeb4f96fb8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal\" (UID: \"7b5b4cb1adc7cdefada5f0eeb4f96fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal" Jul 06 15:26:19.093742 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.093726 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5b4cb1adc7cdefada5f0eeb4f96fb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal\" (UID: \"7b5b4cb1adc7cdefada5f0eeb4f96fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal" Jul 06 15:26:19.093774 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.093751 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91375dee49182778f247ddc7d3913f13-config\") pod \"kube-apiserver-proxy-ip-10-0-129-225.ec2.internal\" (UID: \"91375dee49182778f247ddc7d3913f13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-225.ec2.internal" Jul 06 15:26:19.093774 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.093770 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/91375dee49182778f247ddc7d3913f13-config\") pod \"kube-apiserver-proxy-ip-10-0-129-225.ec2.internal\" (UID: \"91375dee49182778f247ddc7d3913f13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-225.ec2.internal" Jul 06 15:26:19.093835 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.093796 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b5b4cb1adc7cdefada5f0eeb4f96fb8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal\" (UID: \"7b5b4cb1adc7cdefada5f0eeb4f96fb8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal" Jul 06 15:26:19.179063 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:19.179009 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-225.ec2.internal\" not found" Jul 06 15:26:19.223442 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.223424 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal" Jul 06 15:26:19.228304 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.228286 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-225.ec2.internal" Jul 06 15:26:19.279756 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:19.279729 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-225.ec2.internal\" not found" Jul 06 15:26:19.380314 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:19.380286 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-225.ec2.internal\" not found" Jul 06 15:26:19.480775 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:19.480751 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-225.ec2.internal\" not found" Jul 06 15:26:19.581328 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:19.581287 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-225.ec2.internal\" not found" Jul 06 15:26:19.588644 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.588625 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:26:19.634485 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.634469 2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 15:26:19.634929 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.634587 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 15:26:19.634929 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.634616 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 15:26:19.681534 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:19.681512 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-225.ec2.internal\" not found" Jul 06 15:26:19.708825 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.708793 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 15:21:19 +0000 UTC" deadline="2028-03-09 22:55:23.266099886 +0000 UTC" Jul 06 15:26:19.708825 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.708817 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14695h29m3.557285714s" Jul 06 15:26:19.781713 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:19.781685 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-225.ec2.internal\" not found" Jul 06 15:26:19.882232 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:19.882191 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-225.ec2.internal\" not found" Jul 06 15:26:19.926744 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.926714 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:26:19.945280 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:19.945249 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91375dee49182778f247ddc7d3913f13.slice/crio-1c595c6cab7bae9f0f6764fee47cc67824c8477e7eca32c76f330f18ede86027 WatchSource:0}: Error finding container 1c595c6cab7bae9f0f6764fee47cc67824c8477e7eca32c76f330f18ede86027: Status 404 returned error can't find the container with id 1c595c6cab7bae9f0f6764fee47cc67824c8477e7eca32c76f330f18ede86027 Jul 06 15:26:19.953282 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.953262 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 15:26:19.959114 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:19.959095 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5b4cb1adc7cdefada5f0eeb4f96fb8.slice/crio-0a8cfd3454f99fb3aba3016bbef124f4e9380e611b410cc9a952a19ed5635051 WatchSource:0}: Error finding container 0a8cfd3454f99fb3aba3016bbef124f4e9380e611b410cc9a952a19ed5635051: Status 404 returned error can't find the container with id 0a8cfd3454f99fb3aba3016bbef124f4e9380e611b410cc9a952a19ed5635051 Jul 06 15:26:19.992149 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:19.992126 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-225.ec2.internal" Jul 06 15:26:20.003774 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.003752 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 15:26:20.005520 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.005503 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal" Jul 06 15:26:20.017334 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.017310 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 15:26:20.055649 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.055580 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:26:20.670184 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.670153 2561 apiserver.go:52] "Watching apiserver" Jul 06 15:26:20.683283 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.682885 2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 15:26:20.684259 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.684225 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal","openshift-multus/multus-vv6s5","openshift-network-operator/iptables-alerter-kxmff","kube-system/global-pull-secret-syncer-77mbr","kube-system/kube-apiserver-proxy-ip-10-0-129-225.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj","openshift-image-registry/node-ca-w44v2","openshift-multus/multus-additional-cni-plugins-m2h7f","openshift-multus/network-metrics-daemon-m4llb","openshift-network-diagnostics/network-check-target-9hpcm","openshift-ovn-kubernetes/ovnkube-node-562c5","kube-system/konnectivity-agent-lhbjw","openshift-cluster-node-tuning-operator/tuned-jwksx","openshift-dns/node-resolver-w8l5z"] Jul 06 15:26:20.687673 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.686901 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w44v2" Jul 06 15:26:20.689442 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.689422 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.690628 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.690334 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 15:26:20.690628 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.690381 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 15:26:20.690628 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.690334 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5s2vc\"" Jul 06 15:26:20.690628 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.690634 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 15:26:20.691958 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.691888 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 15:26:20.692972 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.692689 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kxmff" Jul 06 15:26:20.694439 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.693327 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 15:26:20.694439 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.693423 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 15:26:20.694439 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.693551 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 15:26:20.694439 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.694425 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 15:26:20.694751 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.694703 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 15:26:20.695160 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.694924 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xfjjd\"" Jul 06 15:26:20.695160 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.694947 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 15:26:20.695687 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.695665 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lk6qz\"" Jul 06 15:26:20.696040 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.696017 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:26:20.696361 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.696130 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 15:26:20.696361 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.696246 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 15:26:20.698908 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.697099 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:20.698908 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:20.697178 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.700848 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w8l5z" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701525 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-cni-netd\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701575 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f61903e-f088-4e01-822c-a13af1322cda-host-slash\") pod \"iptables-alerter-kxmff\" (UID: \"3f61903e-f088-4e01-822c-a13af1322cda\") " pod="openshift-network-operator/iptables-alerter-kxmff" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701606 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l297\" (UniqueName: \"kubernetes.io/projected/3f61903e-f088-4e01-822c-a13af1322cda-kube-api-access-8l297\") pod \"iptables-alerter-kxmff\" (UID: \"3f61903e-f088-4e01-822c-a13af1322cda\") " pod="openshift-network-operator/iptables-alerter-kxmff" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701632 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjdgc\" (UniqueName: \"kubernetes.io/projected/3b2617bf-d392-4486-ab8c-3d0aa5f481aa-kube-api-access-vjdgc\") pod \"node-ca-w44v2\" (UID: \"3b2617bf-d392-4486-ab8c-3d0aa5f481aa\") " pod="openshift-image-registry/node-ca-w44v2" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701658 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-run-ovn\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701682 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87582e94-73f6-4fc5-bcd8-07fda52170b5-env-overrides\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701709 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701734 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-kubelet\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701759 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-slash\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701783 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-run-systemd\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701807 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-node-log\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701839 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-log-socket\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701863 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87582e94-73f6-4fc5-bcd8-07fda52170b5-ovnkube-script-lib\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701916 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701944 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-systemd-units\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.703563 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701968 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-run-netns\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.701992 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-etc-openvswitch\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.702023 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-cni-bin\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.702054 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87582e94-73f6-4fc5-bcd8-07fda52170b5-ovnkube-config\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.702080 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4c84634-a9f3-4cc5-a67b-09805e8f67af-dbus\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.702106 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b2617bf-d392-4486-ab8c-3d0aa5f481aa-serviceca\") pod \"node-ca-w44v2\" (UID: \"3b2617bf-d392-4486-ab8c-3d0aa5f481aa\") " pod="openshift-image-registry/node-ca-w44v2" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.702133 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-run-openvswitch\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.702157 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b2617bf-d392-4486-ab8c-3d0aa5f481aa-host\") pod \"node-ca-w44v2\" (UID: \"3b2617bf-d392-4486-ab8c-3d0aa5f481aa\") " pod="openshift-image-registry/node-ca-w44v2" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.702182 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-var-lib-openvswitch\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.702207 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-run-ovn-kubernetes\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.702238 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfj9f\" (UniqueName: \"kubernetes.io/projected/87582e94-73f6-4fc5-bcd8-07fda52170b5-kube-api-access-tfj9f\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.702265 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f61903e-f088-4e01-822c-a13af1322cda-iptables-alerter-script\") pod \"iptables-alerter-kxmff\" (UID: \"3f61903e-f088-4e01-822c-a13af1322cda\") " pod="openshift-network-operator/iptables-alerter-kxmff" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.702291 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4c84634-a9f3-4cc5-a67b-09805e8f67af-kubelet-config\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.702343 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87582e94-73f6-4fc5-bcd8-07fda52170b5-ovn-node-metrics-cert\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.703792 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dv65k\"" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.703828 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.703851 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 15:26:20.704466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.704125 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.706914 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.706765 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:20.706914 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:20.706855 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:20.708759 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.707814 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 15:26:20.708759 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.707855 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 15:26:20.708759 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.708114 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 15:26:20.708759 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.708352 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rcvj7\"" Jul 06 15:26:20.708759 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.708630 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 15:26:20.709621 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.709582 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.712270 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.712233 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.712980 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.712841 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 15:26:20.716789 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.712841 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:26:20.719505 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.719480 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 15:26:20.719740 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.719715 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 15:26:20.719974 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.719960 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 15:26:20.720172 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.720158 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 15:26:20.720292 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.719713 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qqt8r\"" Jul 06 15:26:20.720899 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.720878 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-94q72\"" Jul 06 15:26:20.721100 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.721076 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:20.721509 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:20.721484 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:20.722713 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.722692 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.725538 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.725514 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lhbjw" Jul 06 15:26:20.726710 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.726688 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6vrcr" Jul 06 15:26:20.726807 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.726767 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 15:26:20.726807 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.726796 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 15:26:20.726924 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.726860 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4ktjr\"" Jul 06 15:26:20.728575 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.728554 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9r2b6\"" Jul 06 15:26:20.728690 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.728567 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 15:26:20.728880 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.728805 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 15:26:20.735471 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.735452 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6vrcr" Jul 06 15:26:20.794227 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.794197 2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 15:26:20.802543 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802509 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4c84634-a9f3-4cc5-a67b-09805e8f67af-dbus\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:20.802669 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802544 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b2617bf-d392-4486-ab8c-3d0aa5f481aa-serviceca\") pod \"node-ca-w44v2\" (UID: \"3b2617bf-d392-4486-ab8c-3d0aa5f481aa\") " pod="openshift-image-registry/node-ca-w44v2" Jul 06 15:26:20.802669 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802576 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-run-openvswitch\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.802669 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802622 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-hostroot\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.802669 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802649 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-var-lib-openvswitch\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.802861 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802684 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-run-ovn-kubernetes\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.802861 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802710 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6-hosts-file\") pod \"node-resolver-w8l5z\" (UID: \"75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6\") " pod="openshift-dns/node-resolver-w8l5z" Jul 06 15:26:20.802861 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802736 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-var-lib-cni-bin\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.802861 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802761 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c70229f0-9a94-4e50-84c6-6c549c7cb2db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.802861 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802787 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79mzd\" (UniqueName: \"kubernetes.io/projected/d047efe8-bfb0-4fee-a069-15a344dc4726-kube-api-access-79mzd\") pod \"network-metrics-daemon-m4llb\" (UID: \"d047efe8-bfb0-4fee-a069-15a344dc4726\") " pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:20.802861 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802810 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-sys-fs\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.802861 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802836 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-multus-cni-dir\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.803204 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802861 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4c84634-a9f3-4cc5-a67b-09805e8f67af-kubelet-config\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:20.803204 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802890 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-etc-selinux\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.803204 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802921 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-var-lib-kubelet\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.803204 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802945 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c70229f0-9a94-4e50-84c6-6c549c7cb2db-cnibin\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.803204 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802969 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-kubernetes\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.803204 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.802991 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/266558dc-b3ed-4050-9950-89db97f0b839-tmp\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.803204 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803015 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6fa66968-290f-4570-b983-15d1d692f9b5-konnectivity-ca\") pod \"konnectivity-agent-lhbjw\" (UID: \"6fa66968-290f-4570-b983-15d1d692f9b5\") " pod="kube-system/konnectivity-agent-lhbjw" Jul 06 15:26:20.803204 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803047 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l297\" (UniqueName: \"kubernetes.io/projected/3f61903e-f088-4e01-822c-a13af1322cda-kube-api-access-8l297\") pod \"iptables-alerter-kxmff\" (UID: \"3f61903e-f088-4e01-822c-a13af1322cda\") " pod="openshift-network-operator/iptables-alerter-kxmff" Jul 06 15:26:20.803204 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803074 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87582e94-73f6-4fc5-bcd8-07fda52170b5-env-overrides\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.803204 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803098 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-socket-dir\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.803204 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803122 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-systemd\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.803204 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803152 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-slash\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.803204 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803177 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-log-socket\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803218 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87582e94-73f6-4fc5-bcd8-07fda52170b5-ovnkube-script-lib\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803242 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-etc-kubernetes\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803262 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-sysctl-conf\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803289 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803330 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqh7v\" (UniqueName: \"kubernetes.io/projected/75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6-kube-api-access-tqh7v\") pod \"node-resolver-w8l5z\" (UID: \"75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6\") " pod="openshift-dns/node-resolver-w8l5z" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803351 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-device-dir\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803377 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-var-lib-kubelet\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803400 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87582e94-73f6-4fc5-bcd8-07fda52170b5-ovnkube-config\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803431 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-var-lib-cni-multus\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803450 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6-tmp-dir\") pod \"node-resolver-w8l5z\" (UID: \"75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6\") " pod="openshift-dns/node-resolver-w8l5z" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803475 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-os-release\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803500 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-multus-socket-dir-parent\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803523 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-run-netns\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803540 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c70229f0-9a94-4e50-84c6-6c549c7cb2db-system-cni-dir\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803561 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c70229f0-9a94-4e50-84c6-6c549c7cb2db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.803766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803580 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkkt8\" (UniqueName: \"kubernetes.io/projected/c70229f0-9a94-4e50-84c6-6c549c7cb2db-kube-api-access-pkkt8\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803597 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs\") pod \"network-metrics-daemon-m4llb\" (UID: \"d047efe8-bfb0-4fee-a069-15a344dc4726\") " pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803633 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b2617bf-d392-4486-ab8c-3d0aa5f481aa-host\") pod \"node-ca-w44v2\" (UID: \"3b2617bf-d392-4486-ab8c-3d0aa5f481aa\") " pod="openshift-image-registry/node-ca-w44v2" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803664 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfj9f\" (UniqueName: \"kubernetes.io/projected/87582e94-73f6-4fc5-bcd8-07fda52170b5-kube-api-access-tfj9f\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803692 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803716 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-multus-conf-dir\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803732 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-sysconfig\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803750 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-sys\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803768 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-lib-modules\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803792 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-host\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803849 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f61903e-f088-4e01-822c-a13af1322cda-iptables-alerter-script\") pod \"iptables-alerter-kxmff\" (UID: \"3f61903e-f088-4e01-822c-a13af1322cda\") " pod="openshift-network-operator/iptables-alerter-kxmff" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803877 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c530b7d-1882-40f2-a756-73891ae3ece4-multus-daemon-config\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803949 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c70229f0-9a94-4e50-84c6-6c549c7cb2db-os-release\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803972 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-modprobe-d\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.803989 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-run\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804019 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtsnt\" (UniqueName: \"kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt\") pod \"network-check-target-9hpcm\" (UID: \"30f35138-27d5-4f8c-ae95-ed63e6de45ae\") " pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804045 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87582e94-73f6-4fc5-bcd8-07fda52170b5-ovn-node-metrics-cert\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.804569 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804060 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-system-cni-dir\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804084 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c70229f0-9a94-4e50-84c6-6c549c7cb2db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804105 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/266558dc-b3ed-4050-9950-89db97f0b839-etc-tuned\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804120 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-cni-netd\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804140 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f61903e-f088-4e01-822c-a13af1322cda-host-slash\") pod \"iptables-alerter-kxmff\" (UID: \"3f61903e-f088-4e01-822c-a13af1322cda\") " pod="openshift-network-operator/iptables-alerter-kxmff" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804169 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjdgc\" (UniqueName: \"kubernetes.io/projected/3b2617bf-d392-4486-ab8c-3d0aa5f481aa-kube-api-access-vjdgc\") pod \"node-ca-w44v2\" (UID: \"3b2617bf-d392-4486-ab8c-3d0aa5f481aa\") " pod="openshift-image-registry/node-ca-w44v2" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804193 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-run-ovn\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804209 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8kdl\" (UniqueName: \"kubernetes.io/projected/8c530b7d-1882-40f2-a756-73891ae3ece4-kube-api-access-m8kdl\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804239 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804266 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-kubelet\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804295 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-run-systemd\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804323 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-node-log\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804350 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwtft\" (UniqueName: \"kubernetes.io/projected/348ff35a-122a-49be-8f2a-2a1dd1032b56-kube-api-access-dwtft\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804377 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c530b7d-1882-40f2-a756-73891ae3ece4-cni-binary-copy\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804404 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-run-k8s-cni-cncf-io\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804432 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-run-multus-certs\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804469 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c70229f0-9a94-4e50-84c6-6c549c7cb2db-cni-binary-copy\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.805410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804496 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-sysctl-d\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804521 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg22x\" (UniqueName: \"kubernetes.io/projected/266558dc-b3ed-4050-9950-89db97f0b839-kube-api-access-gg22x\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804549 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6fa66968-290f-4570-b983-15d1d692f9b5-agent-certs\") pod \"konnectivity-agent-lhbjw\" (UID: \"6fa66968-290f-4570-b983-15d1d692f9b5\") " pod="kube-system/konnectivity-agent-lhbjw" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804581 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-systemd-units\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804625 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-run-netns\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804656 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-etc-openvswitch\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804683 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-cni-bin\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804711 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-registration-dir\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804738 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-cnibin\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.804849 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4c84634-a9f3-4cc5-a67b-09805e8f67af-dbus\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.805188 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b2617bf-d392-4486-ab8c-3d0aa5f481aa-serviceca\") pod \"node-ca-w44v2\" (UID: \"3b2617bf-d392-4486-ab8c-3d0aa5f481aa\") " pod="openshift-image-registry/node-ca-w44v2" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.805244 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-run-openvswitch\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.805282 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-var-lib-openvswitch\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.805338 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-run-ovn-kubernetes\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.805423 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4c84634-a9f3-4cc5-a67b-09805e8f67af-kubelet-config\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.806106 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87582e94-73f6-4fc5-bcd8-07fda52170b5-env-overrides\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.806174 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-slash\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.806230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.806205 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-log-socket\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.806992 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.806549 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87582e94-73f6-4fc5-bcd8-07fda52170b5-ovnkube-script-lib\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.806992 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.806658 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.806992 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.806752 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal" event={"ID":"7b5b4cb1adc7cdefada5f0eeb4f96fb8","Type":"ContainerStarted","Data":"0a8cfd3454f99fb3aba3016bbef124f4e9380e611b410cc9a952a19ed5635051"} Jul 06 15:26:20.806992 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.806971 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b2617bf-d392-4486-ab8c-3d0aa5f481aa-host\") pod \"node-ca-w44v2\" (UID: \"3b2617bf-d392-4486-ab8c-3d0aa5f481aa\") " pod="openshift-image-registry/node-ca-w44v2" Jul 06 15:26:20.807199 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.807136 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87582e94-73f6-4fc5-bcd8-07fda52170b5-ovnkube-config\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.807256 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.807238 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-cni-netd\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.807339 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.807317 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f61903e-f088-4e01-822c-a13af1322cda-host-slash\") pod \"iptables-alerter-kxmff\" (UID: \"3f61903e-f088-4e01-822c-a13af1322cda\") " pod="openshift-network-operator/iptables-alerter-kxmff" Jul 06 15:26:20.807397 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.807321 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-run-ovn\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.807451 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:20.807424 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:26:20.807503 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:20.807491 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret podName:b4c84634-a9f3-4cc5-a67b-09805e8f67af nodeName:}" failed. No retries permitted until 2026-07-06 15:26:21.307468625 +0000 UTC m=+2.940020610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret") pod "global-pull-secret-syncer-77mbr" (UID: "b4c84634-a9f3-4cc5-a67b-09805e8f67af") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:26:20.807700 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.807682 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-run-netns\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.807787 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.807677 2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 15:26:20.807787 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.807732 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-systemd-units\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.807787 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.807744 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-run-systemd\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.807787 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.807744 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-etc-openvswitch\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.807787 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.807769 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-cni-bin\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.808293 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.807821 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f61903e-f088-4e01-822c-a13af1322cda-iptables-alerter-script\") pod \"iptables-alerter-kxmff\" (UID: \"3f61903e-f088-4e01-822c-a13af1322cda\") " pod="openshift-network-operator/iptables-alerter-kxmff" Jul 06 15:26:20.808293 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.807856 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-host-kubelet\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.808293 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.807903 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87582e94-73f6-4fc5-bcd8-07fda52170b5-node-log\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.810046 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.810019 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-225.ec2.internal" event={"ID":"91375dee49182778f247ddc7d3913f13","Type":"ContainerStarted","Data":"1c595c6cab7bae9f0f6764fee47cc67824c8477e7eca32c76f330f18ede86027"} Jul 06 15:26:20.814964 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.811804 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87582e94-73f6-4fc5-bcd8-07fda52170b5-ovn-node-metrics-cert\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.816153 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.816132 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l297\" (UniqueName: \"kubernetes.io/projected/3f61903e-f088-4e01-822c-a13af1322cda-kube-api-access-8l297\") pod \"iptables-alerter-kxmff\" (UID: \"3f61903e-f088-4e01-822c-a13af1322cda\") " pod="openshift-network-operator/iptables-alerter-kxmff" Jul 06 15:26:20.817479 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.817450 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfj9f\" (UniqueName: \"kubernetes.io/projected/87582e94-73f6-4fc5-bcd8-07fda52170b5-kube-api-access-tfj9f\") pod \"ovnkube-node-562c5\" (UID: \"87582e94-73f6-4fc5-bcd8-07fda52170b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:20.818260 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.818238 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjdgc\" (UniqueName: \"kubernetes.io/projected/3b2617bf-d392-4486-ab8c-3d0aa5f481aa-kube-api-access-vjdgc\") pod \"node-ca-w44v2\" (UID: \"3b2617bf-d392-4486-ab8c-3d0aa5f481aa\") " pod="openshift-image-registry/node-ca-w44v2" Jul 06 15:26:20.905901 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.905847 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-lib-modules\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.906135 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.905915 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-host\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.906135 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.905944 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c530b7d-1882-40f2-a756-73891ae3ece4-multus-daemon-config\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.906135 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.905967 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c70229f0-9a94-4e50-84c6-6c549c7cb2db-os-release\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.906135 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.905989 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-modprobe-d\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.906135 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906012 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-run\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.906135 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906037 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtsnt\" (UniqueName: \"kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt\") pod \"network-check-target-9hpcm\" (UID: \"30f35138-27d5-4f8c-ae95-ed63e6de45ae\") " pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:20.906135 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906046 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-lib-modules\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.906135 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906063 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-system-cni-dir\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.906135 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906097 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c70229f0-9a94-4e50-84c6-6c549c7cb2db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.906135 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906119 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-system-cni-dir\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.906135 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906126 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/266558dc-b3ed-4050-9950-89db97f0b839-etc-tuned\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906158 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8kdl\" (UniqueName: \"kubernetes.io/projected/8c530b7d-1882-40f2-a756-73891ae3ece4-kube-api-access-m8kdl\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906170 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-host\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906205 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwtft\" (UniqueName: \"kubernetes.io/projected/348ff35a-122a-49be-8f2a-2a1dd1032b56-kube-api-access-dwtft\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906229 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c530b7d-1882-40f2-a756-73891ae3ece4-cni-binary-copy\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906261 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-run-k8s-cni-cncf-io\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906287 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-run-multus-certs\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906310 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c70229f0-9a94-4e50-84c6-6c549c7cb2db-cni-binary-copy\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906335 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-sysctl-d\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906360 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg22x\" (UniqueName: \"kubernetes.io/projected/266558dc-b3ed-4050-9950-89db97f0b839-kube-api-access-gg22x\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906387 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6fa66968-290f-4570-b983-15d1d692f9b5-agent-certs\") pod \"konnectivity-agent-lhbjw\" (UID: \"6fa66968-290f-4570-b983-15d1d692f9b5\") " pod="kube-system/konnectivity-agent-lhbjw" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906417 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-registration-dir\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906441 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-cnibin\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906473 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-hostroot\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906500 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6-hosts-file\") pod \"node-resolver-w8l5z\" (UID: \"75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6\") " pod="openshift-dns/node-resolver-w8l5z" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906525 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-var-lib-cni-bin\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906552 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c70229f0-9a94-4e50-84c6-6c549c7cb2db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.906724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906603 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mzd\" (UniqueName: \"kubernetes.io/projected/d047efe8-bfb0-4fee-a069-15a344dc4726-kube-api-access-79mzd\") pod \"network-metrics-daemon-m4llb\" (UID: \"d047efe8-bfb0-4fee-a069-15a344dc4726\") " pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906632 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-sys-fs\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906740 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c70229f0-9a94-4e50-84c6-6c549c7cb2db-os-release\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906760 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-modprobe-d\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906809 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-multus-cni-dir\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906814 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-sys-fs\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906845 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-etc-selinux\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906854 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8c530b7d-1882-40f2-a756-73891ae3ece4-multus-daemon-config\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906873 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-var-lib-kubelet\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906902 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c70229f0-9a94-4e50-84c6-6c549c7cb2db-cnibin\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906929 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-kubernetes\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906955 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/266558dc-b3ed-4050-9950-89db97f0b839-tmp\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906981 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6fa66968-290f-4570-b983-15d1d692f9b5-konnectivity-ca\") pod \"konnectivity-agent-lhbjw\" (UID: \"6fa66968-290f-4570-b983-15d1d692f9b5\") " pod="kube-system/konnectivity-agent-lhbjw" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907011 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-socket-dir\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907038 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-systemd\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907068 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-etc-kubernetes\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907093 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-sysctl-conf\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.907767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907122 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqh7v\" (UniqueName: \"kubernetes.io/projected/75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6-kube-api-access-tqh7v\") pod \"node-resolver-w8l5z\" (UID: \"75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6\") " pod="openshift-dns/node-resolver-w8l5z" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907154 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-device-dir\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907165 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-registration-dir\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907180 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-var-lib-kubelet\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907207 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-var-lib-cni-multus\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907216 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c70229f0-9a94-4e50-84c6-6c549c7cb2db-cnibin\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907229 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-systemd\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907221 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-multus-cni-dir\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907232 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6-tmp-dir\") pod \"node-resolver-w8l5z\" (UID: \"75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6\") " pod="openshift-dns/node-resolver-w8l5z" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907280 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-os-release\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907284 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-run-multus-certs\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907314 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-multus-socket-dir-parent\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907318 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8c530b7d-1882-40f2-a756-73891ae3ece4-cni-binary-copy\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907341 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-device-dir\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907345 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-run-netns\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907370 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-sysctl-conf\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907392 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-run-netns\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907405 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6-hosts-file\") pod \"node-resolver-w8l5z\" (UID: \"75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6\") " pod="openshift-dns/node-resolver-w8l5z" Jul 06 15:26:20.908803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.906929 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-run\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907437 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c70229f0-9a94-4e50-84c6-6c549c7cb2db-system-cni-dir\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907471 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c70229f0-9a94-4e50-84c6-6c549c7cb2db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907502 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkkt8\" (UniqueName: \"kubernetes.io/projected/c70229f0-9a94-4e50-84c6-6c549c7cb2db-kube-api-access-pkkt8\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907594 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-var-lib-kubelet\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907645 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs\") pod \"network-metrics-daemon-m4llb\" (UID: \"d047efe8-bfb0-4fee-a069-15a344dc4726\") " pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907680 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907707 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-multus-conf-dir\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907732 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-sysconfig\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907757 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-sys\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907780 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c70229f0-9a94-4e50-84c6-6c549c7cb2db-cni-binary-copy\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907786 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-cnibin\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907842 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:20.907896 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907889 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-sys\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907934 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-socket-dir\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:20.907959 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs podName:d047efe8-bfb0-4fee-a069-15a344dc4726 nodeName:}" failed. No retries permitted until 2026-07-06 15:26:21.407939795 +0000 UTC m=+3.040491760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs") pod "network-metrics-daemon-m4llb" (UID: "d047efe8-bfb0-4fee-a069-15a344dc4726") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:26:20.909667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.908014 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-var-lib-cni-multus\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907959 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-var-lib-cni-bin\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.908015 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-os-release\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.908069 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-multus-socket-dir-parent\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.908076 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-etc-kubernetes\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907789 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6-tmp-dir\") pod \"node-resolver-w8l5z\" (UID: \"75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6\") " pod="openshift-dns/node-resolver-w8l5z" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.908119 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c70229f0-9a94-4e50-84c6-6c549c7cb2db-system-cni-dir\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.908167 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-hostroot\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907957 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c70229f0-9a94-4e50-84c6-6c549c7cb2db-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.908416 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-var-lib-kubelet\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.907161 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-kubernetes\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.908444 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-sysconfig\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.908468 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6fa66968-290f-4570-b983-15d1d692f9b5-konnectivity-ca\") pod \"konnectivity-agent-lhbjw\" (UID: \"6fa66968-290f-4570-b983-15d1d692f9b5\") " pod="kube-system/konnectivity-agent-lhbjw" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.908481 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-host-run-k8s-cni-cncf-io\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.908516 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8c530b7d-1882-40f2-a756-73891ae3ece4-multus-conf-dir\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.908649 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/266558dc-b3ed-4050-9950-89db97f0b839-etc-sysctl-d\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.908694 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/348ff35a-122a-49be-8f2a-2a1dd1032b56-etc-selinux\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.908815 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c70229f0-9a94-4e50-84c6-6c549c7cb2db-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.910564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.909077 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c70229f0-9a94-4e50-84c6-6c549c7cb2db-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.911508 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.911139 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6fa66968-290f-4570-b983-15d1d692f9b5-agent-certs\") pod \"konnectivity-agent-lhbjw\" (UID: \"6fa66968-290f-4570-b983-15d1d692f9b5\") " pod="kube-system/konnectivity-agent-lhbjw" Jul 06 15:26:20.911508 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.911455 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/266558dc-b3ed-4050-9950-89db97f0b839-etc-tuned\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.911972 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.911950 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/266558dc-b3ed-4050-9950-89db97f0b839-tmp\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:20.920114 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:20.917634 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:26:20.920114 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:20.917659 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:26:20.920114 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:20.917676 2561 projected.go:196] Error preparing data for projected volume kube-api-access-mtsnt for pod openshift-network-diagnostics/network-check-target-9hpcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:26:20.920114 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:20.917739 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt podName:30f35138-27d5-4f8c-ae95-ed63e6de45ae nodeName:}" failed. No retries permitted until 2026-07-06 15:26:21.417721112 +0000 UTC m=+3.050273096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mtsnt" (UniqueName: "kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt") pod "network-check-target-9hpcm" (UID: "30f35138-27d5-4f8c-ae95-ed63e6de45ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:26:20.921000 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.920925 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkkt8\" (UniqueName: \"kubernetes.io/projected/c70229f0-9a94-4e50-84c6-6c549c7cb2db-kube-api-access-pkkt8\") pod \"multus-additional-cni-plugins-m2h7f\" (UID: \"c70229f0-9a94-4e50-84c6-6c549c7cb2db\") " pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:20.923954 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.922135 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwtft\" (UniqueName: \"kubernetes.io/projected/348ff35a-122a-49be-8f2a-2a1dd1032b56-kube-api-access-dwtft\") pod \"aws-ebs-csi-driver-node-kwvxj\" (UID: \"348ff35a-122a-49be-8f2a-2a1dd1032b56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:20.923954 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.923757 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqh7v\" (UniqueName: \"kubernetes.io/projected/75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6-kube-api-access-tqh7v\") pod \"node-resolver-w8l5z\" (UID: \"75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6\") " pod="openshift-dns/node-resolver-w8l5z" Jul 06 15:26:20.923954 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.923929 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8kdl\" (UniqueName: \"kubernetes.io/projected/8c530b7d-1882-40f2-a756-73891ae3ece4-kube-api-access-m8kdl\") pod \"multus-vv6s5\" (UID: \"8c530b7d-1882-40f2-a756-73891ae3ece4\") " pod="openshift-multus/multus-vv6s5" Jul 06 15:26:20.925487 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.925462 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mzd\" (UniqueName: \"kubernetes.io/projected/d047efe8-bfb0-4fee-a069-15a344dc4726-kube-api-access-79mzd\") pod \"network-metrics-daemon-m4llb\" (UID: \"d047efe8-bfb0-4fee-a069-15a344dc4726\") " pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:20.925595 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:20.925535 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg22x\" (UniqueName: \"kubernetes.io/projected/266558dc-b3ed-4050-9950-89db97f0b839-kube-api-access-gg22x\") pod \"tuned-jwksx\" (UID: \"266558dc-b3ed-4050-9950-89db97f0b839\") " pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:21.004953 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.004910 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w44v2" Jul 06 15:26:21.014967 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:21.014943 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2617bf_d392_4486_ab8c_3d0aa5f481aa.slice/crio-e4cf0d8875b42ef8d9d4a48fab9579a49b8b198f688d340599c2d1124d03c7f3 WatchSource:0}: Error finding container e4cf0d8875b42ef8d9d4a48fab9579a49b8b198f688d340599c2d1124d03c7f3: Status 404 returned error can't find the container with id e4cf0d8875b42ef8d9d4a48fab9579a49b8b198f688d340599c2d1124d03c7f3 Jul 06 15:26:21.027744 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.027720 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:21.038182 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:21.038156 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87582e94_73f6_4fc5_bcd8_07fda52170b5.slice/crio-d17c85b3c9c811e1fe6a0f7a7cc903f5b83ca1b1f220e433dacd9763f7e75b16 WatchSource:0}: Error finding container d17c85b3c9c811e1fe6a0f7a7cc903f5b83ca1b1f220e433dacd9763f7e75b16: Status 404 returned error can't find the container with id d17c85b3c9c811e1fe6a0f7a7cc903f5b83ca1b1f220e433dacd9763f7e75b16 Jul 06 15:26:21.039118 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.039096 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kxmff" Jul 06 15:26:21.045356 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.045335 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w8l5z" Jul 06 15:26:21.050736 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:21.050709 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f61903e_f088_4e01_822c_a13af1322cda.slice/crio-5099a0805a7fed7529d39fb05eaf15380bb0304e4467c6da734ab001111e9e56 WatchSource:0}: Error finding container 5099a0805a7fed7529d39fb05eaf15380bb0304e4467c6da734ab001111e9e56: Status 404 returned error can't find the container with id 5099a0805a7fed7529d39fb05eaf15380bb0304e4467c6da734ab001111e9e56 Jul 06 15:26:21.052759 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.052740 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" Jul 06 15:26:21.062140 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.062116 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jwksx" Jul 06 15:26:21.063574 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:21.063549 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348ff35a_122a_49be_8f2a_2a1dd1032b56.slice/crio-229ce1da0ff535adbe7dbd91284299a0bbea14415c8b8022b9550657478d8e54 WatchSource:0}: Error finding container 229ce1da0ff535adbe7dbd91284299a0bbea14415c8b8022b9550657478d8e54: Status 404 returned error can't find the container with id 229ce1da0ff535adbe7dbd91284299a0bbea14415c8b8022b9550657478d8e54 Jul 06 15:26:21.071199 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.071176 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vv6s5" Jul 06 15:26:21.074132 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:21.074105 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266558dc_b3ed_4050_9950_89db97f0b839.slice/crio-d2a5cd632df609dd3b261461f79c2f96a52c964474edfad9355c65473d242584 WatchSource:0}: Error finding container d2a5cd632df609dd3b261461f79c2f96a52c964474edfad9355c65473d242584: Status 404 returned error can't find the container with id d2a5cd632df609dd3b261461f79c2f96a52c964474edfad9355c65473d242584 Jul 06 15:26:21.079153 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.079131 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m2h7f" Jul 06 15:26:21.081350 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:21.081322 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c530b7d_1882_40f2_a756_73891ae3ece4.slice/crio-2d61659f953cbc1277279bad5750954bc90eefde476de333d6b86bd2c29bea7f WatchSource:0}: Error finding container 2d61659f953cbc1277279bad5750954bc90eefde476de333d6b86bd2c29bea7f: Status 404 returned error can't find the container with id 2d61659f953cbc1277279bad5750954bc90eefde476de333d6b86bd2c29bea7f Jul 06 15:26:21.090490 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:21.090466 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70229f0_9a94_4e50_84c6_6c549c7cb2db.slice/crio-4cffa04be1c0d38fb7f85b7ad38faf2c003ae4e7586168768458b100c6f2c9dc WatchSource:0}: Error finding container 4cffa04be1c0d38fb7f85b7ad38faf2c003ae4e7586168768458b100c6f2c9dc: Status 404 returned error can't find the container with id 4cffa04be1c0d38fb7f85b7ad38faf2c003ae4e7586168768458b100c6f2c9dc Jul 06 15:26:21.093659 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.093475 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lhbjw" Jul 06 15:26:21.104807 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:21.104782 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa66968_290f_4570_b983_15d1d692f9b5.slice/crio-5d1e9742fd74bdf1d810915dab7269e76583aa24600b724fa7a434fcb9afe58c WatchSource:0}: Error finding container 5d1e9742fd74bdf1d810915dab7269e76583aa24600b724fa7a434fcb9afe58c: Status 404 returned error can't find the container with id 5d1e9742fd74bdf1d810915dab7269e76583aa24600b724fa7a434fcb9afe58c Jul 06 15:26:21.140551 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.140530 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:26:21.310567 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.310523 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:21.310741 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:21.310700 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:26:21.310806 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:21.310775 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret podName:b4c84634-a9f3-4cc5-a67b-09805e8f67af nodeName:}" failed. No retries permitted until 2026-07-06 15:26:22.310754368 +0000 UTC m=+3.943306355 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret") pod "global-pull-secret-syncer-77mbr" (UID: "b4c84634-a9f3-4cc5-a67b-09805e8f67af") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:26:21.411965 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.411246 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs\") pod \"network-metrics-daemon-m4llb\" (UID: \"d047efe8-bfb0-4fee-a069-15a344dc4726\") " pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:21.411965 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:21.411384 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:26:21.411965 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:21.411435 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs podName:d047efe8-bfb0-4fee-a069-15a344dc4726 nodeName:}" failed. No retries permitted until 2026-07-06 15:26:22.411419406 +0000 UTC m=+4.043971372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs") pod "network-metrics-daemon-m4llb" (UID: "d047efe8-bfb0-4fee-a069-15a344dc4726") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:26:21.511993 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.511955 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtsnt\" (UniqueName: \"kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt\") pod \"network-check-target-9hpcm\" (UID: \"30f35138-27d5-4f8c-ae95-ed63e6de45ae\") " pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:21.512152 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:21.512104 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:26:21.512152 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:21.512135 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:26:21.512152 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:21.512151 2561 projected.go:196] Error preparing data for projected volume kube-api-access-mtsnt for pod openshift-network-diagnostics/network-check-target-9hpcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:26:21.512363 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:21.512205 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt podName:30f35138-27d5-4f8c-ae95-ed63e6de45ae nodeName:}" failed. No retries permitted until 2026-07-06 15:26:22.512190416 +0000 UTC m=+4.144742378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtsnt" (UniqueName: "kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt") pod "network-check-target-9hpcm" (UID: "30f35138-27d5-4f8c-ae95-ed63e6de45ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:26:21.739346 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.739300 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 15:21:21 +0000 UTC" deadline="2027-12-15 10:00:58.508438725 +0000 UTC" Jul 06 15:26:21.739346 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.739341 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12642h34m36.769102828s" Jul 06 15:26:21.813157 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.813122 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kxmff" event={"ID":"3f61903e-f088-4e01-822c-a13af1322cda","Type":"ContainerStarted","Data":"5099a0805a7fed7529d39fb05eaf15380bb0304e4467c6da734ab001111e9e56"} Jul 06 15:26:21.814877 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.814845 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" event={"ID":"87582e94-73f6-4fc5-bcd8-07fda52170b5","Type":"ContainerStarted","Data":"d17c85b3c9c811e1fe6a0f7a7cc903f5b83ca1b1f220e433dacd9763f7e75b16"} Jul 06 15:26:21.816358 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.816327 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2h7f" event={"ID":"c70229f0-9a94-4e50-84c6-6c549c7cb2db","Type":"ContainerStarted","Data":"4cffa04be1c0d38fb7f85b7ad38faf2c003ae4e7586168768458b100c6f2c9dc"} Jul 06 15:26:21.818303 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.818261 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jwksx" event={"ID":"266558dc-b3ed-4050-9950-89db97f0b839","Type":"ContainerStarted","Data":"d2a5cd632df609dd3b261461f79c2f96a52c964474edfad9355c65473d242584"} Jul 06 15:26:21.820010 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.819981 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" event={"ID":"348ff35a-122a-49be-8f2a-2a1dd1032b56","Type":"ContainerStarted","Data":"229ce1da0ff535adbe7dbd91284299a0bbea14415c8b8022b9550657478d8e54"} Jul 06 15:26:21.821213 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.821186 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w44v2" event={"ID":"3b2617bf-d392-4486-ab8c-3d0aa5f481aa","Type":"ContainerStarted","Data":"e4cf0d8875b42ef8d9d4a48fab9579a49b8b198f688d340599c2d1124d03c7f3"} Jul 06 15:26:21.822882 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.822847 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lhbjw" event={"ID":"6fa66968-290f-4570-b983-15d1d692f9b5","Type":"ContainerStarted","Data":"5d1e9742fd74bdf1d810915dab7269e76583aa24600b724fa7a434fcb9afe58c"} Jul 06 15:26:21.824375 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.824349 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vv6s5" event={"ID":"8c530b7d-1882-40f2-a756-73891ae3ece4","Type":"ContainerStarted","Data":"2d61659f953cbc1277279bad5750954bc90eefde476de333d6b86bd2c29bea7f"} Jul 06 15:26:21.825720 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:21.825697 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w8l5z" event={"ID":"75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6","Type":"ContainerStarted","Data":"1d2ca45597127a8a7655f5ba6280f1b204253703ba8d32e92b05df63862ef4e6"} Jul 06 15:26:22.318978 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:22.318936 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:22.319173 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:22.319096 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:26:22.319173 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:22.319170 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret podName:b4c84634-a9f3-4cc5-a67b-09805e8f67af nodeName:}" failed. No retries permitted until 2026-07-06 15:26:24.319146577 +0000 UTC m=+5.951698557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret") pod "global-pull-secret-syncer-77mbr" (UID: "b4c84634-a9f3-4cc5-a67b-09805e8f67af") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:26:22.420621 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:22.419950 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs\") pod \"network-metrics-daemon-m4llb\" (UID: \"d047efe8-bfb0-4fee-a069-15a344dc4726\") " pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:22.420621 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:22.420146 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:26:22.420621 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:22.420208 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs podName:d047efe8-bfb0-4fee-a069-15a344dc4726 nodeName:}" failed. No retries permitted until 2026-07-06 15:26:24.420190194 +0000 UTC m=+6.052742162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs") pod "network-metrics-daemon-m4llb" (UID: "d047efe8-bfb0-4fee-a069-15a344dc4726") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:26:22.520705 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:22.520666 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtsnt\" (UniqueName: \"kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt\") pod \"network-check-target-9hpcm\" (UID: \"30f35138-27d5-4f8c-ae95-ed63e6de45ae\") " pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:22.520960 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:22.520850 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:26:22.520960 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:22.520871 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:26:22.520960 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:22.520883 2561 projected.go:196] Error preparing data for projected volume kube-api-access-mtsnt for pod openshift-network-diagnostics/network-check-target-9hpcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:26:22.520960 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:22.520939 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt podName:30f35138-27d5-4f8c-ae95-ed63e6de45ae nodeName:}" failed. No retries permitted until 2026-07-06 15:26:24.52092234 +0000 UTC m=+6.153474318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtsnt" (UniqueName: "kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt") pod "network-check-target-9hpcm" (UID: "30f35138-27d5-4f8c-ae95-ed63e6de45ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:26:22.633989 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:22.633910 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:26:22.739677 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:22.739600 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 15:21:21 +0000 UTC" deadline="2028-01-04 16:11:44.956381924 +0000 UTC" Jul 06 15:26:22.739677 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:22.739642 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13128h45m22.216744628s" Jul 06 15:26:22.795671 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:22.795119 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:22.795671 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:22.795126 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:22.795671 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:22.795260 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:22.795671 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:22.795290 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:22.795671 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:22.795401 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:22.795671 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:22.795485 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:23.851243 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:23.850462 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-225.ec2.internal" event={"ID":"91375dee49182778f247ddc7d3913f13","Type":"ContainerStarted","Data":"c6fddc91d7c875c7a17c91b99edd0d1f2b761192d9bc4c48fe9f0c9d0e0c48e1"} Jul 06 15:26:24.337199 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:24.337161 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:24.337393 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:24.337316 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:26:24.337393 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:24.337382 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret podName:b4c84634-a9f3-4cc5-a67b-09805e8f67af nodeName:}" failed. No retries permitted until 2026-07-06 15:26:28.337362377 +0000 UTC m=+9.969914346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret") pod "global-pull-secret-syncer-77mbr" (UID: "b4c84634-a9f3-4cc5-a67b-09805e8f67af") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:26:24.438232 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:24.438191 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs\") pod \"network-metrics-daemon-m4llb\" (UID: \"d047efe8-bfb0-4fee-a069-15a344dc4726\") " pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:24.438433 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:24.438380 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:26:24.438504 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:24.438445 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs podName:d047efe8-bfb0-4fee-a069-15a344dc4726 nodeName:}" failed. No retries permitted until 2026-07-06 15:26:28.438425068 +0000 UTC m=+10.070977038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs") pod "network-metrics-daemon-m4llb" (UID: "d047efe8-bfb0-4fee-a069-15a344dc4726") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:26:24.538688 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:24.538648 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtsnt\" (UniqueName: \"kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt\") pod \"network-check-target-9hpcm\" (UID: \"30f35138-27d5-4f8c-ae95-ed63e6de45ae\") " pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:24.538948 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:24.538905 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:26:24.538948 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:24.538929 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:26:24.538948 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:24.538945 2561 projected.go:196] Error preparing data for projected volume kube-api-access-mtsnt for pod openshift-network-diagnostics/network-check-target-9hpcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:26:24.539138 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:24.539020 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt podName:30f35138-27d5-4f8c-ae95-ed63e6de45ae nodeName:}" failed. No retries permitted until 2026-07-06 15:26:28.539002077 +0000 UTC m=+10.171554046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtsnt" (UniqueName: "kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt") pod "network-check-target-9hpcm" (UID: "30f35138-27d5-4f8c-ae95-ed63e6de45ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:26:24.794749 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:24.794717 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:24.794924 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:24.794850 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:24.795313 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:24.795258 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:24.795438 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:24.795367 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:24.795583 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:24.795567 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:24.795711 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:24.795655 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:24.862804 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:24.862766 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal" event={"ID":"7b5b4cb1adc7cdefada5f0eeb4f96fb8","Type":"ContainerStarted","Data":"82bf6a3522094dc201e6d65190d3b0b395a2335878bbb8df84a67e2ed6ecc34a"} Jul 06 15:26:24.880661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:24.880579 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-225.ec2.internal" podStartSLOduration=4.8805593179999995 podStartE2EDuration="4.880559318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:26:23.871097651 +0000 UTC m=+5.503649639" watchObservedRunningTime="2026-07-06 15:26:24.880559318 +0000 UTC m=+6.513111305" Jul 06 15:26:26.794979 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:26.794709 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:26.794979 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:26.794714 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:26.794979 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:26.794867 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:26.794979 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:26.794880 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:26.794979 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:26.794971 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:26.795721 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:26.795046 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:27.871160 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:27.871121 2561 generic.go:356] "Generic (PLEG): container finished" podID="7b5b4cb1adc7cdefada5f0eeb4f96fb8" containerID="82bf6a3522094dc201e6d65190d3b0b395a2335878bbb8df84a67e2ed6ecc34a" exitCode=0 Jul 06 15:26:27.871671 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:27.871179 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal" event={"ID":"7b5b4cb1adc7cdefada5f0eeb4f96fb8","Type":"ContainerDied","Data":"82bf6a3522094dc201e6d65190d3b0b395a2335878bbb8df84a67e2ed6ecc34a"} Jul 06 15:26:28.371975 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:28.371935 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:28.372161 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:28.372109 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:26:28.372217 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:28.372171 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret podName:b4c84634-a9f3-4cc5-a67b-09805e8f67af nodeName:}" failed. No retries permitted until 2026-07-06 15:26:36.372149204 +0000 UTC m=+18.004701177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret") pod "global-pull-secret-syncer-77mbr" (UID: "b4c84634-a9f3-4cc5-a67b-09805e8f67af") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:26:28.473242 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:28.473203 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs\") pod \"network-metrics-daemon-m4llb\" (UID: \"d047efe8-bfb0-4fee-a069-15a344dc4726\") " pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:28.473429 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:28.473382 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:26:28.473492 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:28.473455 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs podName:d047efe8-bfb0-4fee-a069-15a344dc4726 nodeName:}" failed. No retries permitted until 2026-07-06 15:26:36.473432754 +0000 UTC m=+18.105984739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs") pod "network-metrics-daemon-m4llb" (UID: "d047efe8-bfb0-4fee-a069-15a344dc4726") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:26:28.574742 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:28.574702 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtsnt\" (UniqueName: \"kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt\") pod \"network-check-target-9hpcm\" (UID: \"30f35138-27d5-4f8c-ae95-ed63e6de45ae\") " pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:28.574942 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:28.574867 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:26:28.574942 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:28.574887 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:26:28.574942 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:28.574900 2561 projected.go:196] Error preparing data for projected volume kube-api-access-mtsnt for pod openshift-network-diagnostics/network-check-target-9hpcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:26:28.575113 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:28.574966 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt podName:30f35138-27d5-4f8c-ae95-ed63e6de45ae nodeName:}" failed. No retries permitted until 2026-07-06 15:26:36.574947962 +0000 UTC m=+18.207499947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtsnt" (UniqueName: "kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt") pod "network-check-target-9hpcm" (UID: "30f35138-27d5-4f8c-ae95-ed63e6de45ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:26:28.794878 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:28.794835 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:28.795070 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:28.794958 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:28.795070 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:28.794975 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:28.795070 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:28.795059 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:28.795248 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:28.795113 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:28.795248 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:28.795225 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:30.793586 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:30.793555 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:30.793586 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:30.793588 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:30.794185 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:30.793555 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:30.794185 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:30.793668 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:30.794185 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:30.793773 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:30.794185 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:30.793844 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:32.793799 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:32.793766 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:32.794223 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:32.793884 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:32.794337 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:32.794320 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:32.794447 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:32.794428 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:32.794843 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:32.794815 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:32.794960 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:32.794940 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:34.794192 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:34.794160 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:34.794620 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:34.794296 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:34.794620 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:34.794312 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:34.794620 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:34.794383 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:34.794620 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:34.794433 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:34.794620 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:34.794488 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:36.438281 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:36.438236 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:36.438854 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:36.438437 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:26:36.438854 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:36.438516 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret podName:b4c84634-a9f3-4cc5-a67b-09805e8f67af nodeName:}" failed. No retries permitted until 2026-07-06 15:26:52.438495811 +0000 UTC m=+34.071047773 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret") pod "global-pull-secret-syncer-77mbr" (UID: "b4c84634-a9f3-4cc5-a67b-09805e8f67af") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:26:36.538972 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:36.538942 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs\") pod \"network-metrics-daemon-m4llb\" (UID: \"d047efe8-bfb0-4fee-a069-15a344dc4726\") " pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:36.539153 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:36.539114 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:26:36.539217 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:36.539190 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs podName:d047efe8-bfb0-4fee-a069-15a344dc4726 nodeName:}" failed. No retries permitted until 2026-07-06 15:26:52.539169167 +0000 UTC m=+34.171721148 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs") pod "network-metrics-daemon-m4llb" (UID: "d047efe8-bfb0-4fee-a069-15a344dc4726") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:26:36.639998 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:36.639961 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtsnt\" (UniqueName: \"kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt\") pod \"network-check-target-9hpcm\" (UID: \"30f35138-27d5-4f8c-ae95-ed63e6de45ae\") " pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:36.640165 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:36.640129 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:26:36.640165 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:36.640152 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:26:36.640165 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:36.640164 2561 projected.go:196] Error preparing data for projected volume kube-api-access-mtsnt for pod openshift-network-diagnostics/network-check-target-9hpcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:26:36.640302 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:36.640216 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt podName:30f35138-27d5-4f8c-ae95-ed63e6de45ae nodeName:}" failed. No retries permitted until 2026-07-06 15:26:52.640200721 +0000 UTC m=+34.272752681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtsnt" (UniqueName: "kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt") pod "network-check-target-9hpcm" (UID: "30f35138-27d5-4f8c-ae95-ed63e6de45ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:26:36.794270 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:36.794233 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:36.794453 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:36.794238 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:36.794453 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:36.794365 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:36.794453 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:36.794364 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:36.794659 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:36.794476 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:36.794659 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:36.794572 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:38.795091 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:38.795058 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:38.795511 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:38.795172 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:38.795511 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:38.795257 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:38.795511 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:38.795349 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:38.795511 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:38.795374 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:38.795511 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:38.795412 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:40.794309 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:40.794237 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:40.794309 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:40.794253 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:40.794818 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:40.794369 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:40.794859 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:40.794817 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:40.794901 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:40.794871 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:40.794970 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:40.794940 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:41.899482 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:41.899131 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" event={"ID":"87582e94-73f6-4fc5-bcd8-07fda52170b5","Type":"ContainerStarted","Data":"7bc2858989719ae6051728ba715f96a45f271ad2cb474c25d059310f62c6468c"} Jul 06 15:26:41.900167 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:41.899596 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" event={"ID":"87582e94-73f6-4fc5-bcd8-07fda52170b5","Type":"ContainerStarted","Data":"3548a923d2f39f15b43b408b9f69653484b07c279247341bdab5f2cf8ff94220"} Jul 06 15:26:41.900652 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:41.900633 2561 generic.go:356] "Generic (PLEG): container finished" podID="c70229f0-9a94-4e50-84c6-6c549c7cb2db" containerID="02acf1d6fad2ee1dcc8af335d1355f23d11e88bb9142e35aebe0ad4e36dc300b" exitCode=0 Jul 06 15:26:41.900735 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:41.900706 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2h7f" event={"ID":"c70229f0-9a94-4e50-84c6-6c549c7cb2db","Type":"ContainerDied","Data":"02acf1d6fad2ee1dcc8af335d1355f23d11e88bb9142e35aebe0ad4e36dc300b"} Jul 06 15:26:41.902510 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:41.902481 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jwksx" event={"ID":"266558dc-b3ed-4050-9950-89db97f0b839","Type":"ContainerStarted","Data":"ab4fa60b33be6ae88c2971f3b0a9d4b07bca38509256402cf41be1acecc5cee3"} Jul 06 15:26:41.904106 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:41.904086 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" event={"ID":"348ff35a-122a-49be-8f2a-2a1dd1032b56","Type":"ContainerStarted","Data":"3ffca8047b0874b6dd182d669905093d1e0b9ef86c457ab34fb41624ff88d1a2"} Jul 06 15:26:41.905446 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:41.905422 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w44v2" event={"ID":"3b2617bf-d392-4486-ab8c-3d0aa5f481aa","Type":"ContainerStarted","Data":"18ba5555a9d58b934b87e8d251a51977a22c800fe1d41b2740537fdab0f2f9cc"} Jul 06 15:26:41.907420 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:41.907392 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal" event={"ID":"7b5b4cb1adc7cdefada5f0eeb4f96fb8","Type":"ContainerStarted","Data":"58435e713980e35733451ac0ed52bb48120c1ebfc4dc665f0deaa0d0b057cc0e"} Jul 06 15:26:41.909475 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:41.909446 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lhbjw" event={"ID":"6fa66968-290f-4570-b983-15d1d692f9b5","Type":"ContainerStarted","Data":"f62876adaabef96f6532e368b6d20edd4ef2d14cc45aa8b4605c6c06d21477f7"} Jul 06 15:26:41.910984 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:41.910965 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vv6s5" event={"ID":"8c530b7d-1882-40f2-a756-73891ae3ece4","Type":"ContainerStarted","Data":"1587d7a756a380ad214c37c932c6aa60791df0036ff59384557aeb09e8300559"} Jul 06 15:26:41.912285 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:41.912266 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w8l5z" event={"ID":"75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6","Type":"ContainerStarted","Data":"439c6583e73070981d824b5eba115fabfeb427003adc9ce22f6f0732c9acfb14"} Jul 06 15:26:41.946637 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:41.946603 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vv6s5" podStartSLOduration=2.543552048 podStartE2EDuration="22.946591098s" totalImagesPullingTime="20.40303905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:26:41.946500037 +0000 UTC m=+23.579052019" watchObservedRunningTime="2026-07-06 15:26:41.946591098 +0000 UTC m=+23.579143082" Jul 06 15:26:41.962413 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:41.962372 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-225.ec2.internal" podStartSLOduration=21.96236176 podStartE2EDuration="21.96236176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:26:41.962135446 +0000 UTC m=+23.594687428" watchObservedRunningTime="2026-07-06 15:26:41.96236176 +0000 UTC m=+23.594913743" Jul 06 15:26:41.976776 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:41.976744 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w44v2" podStartSLOduration=2.567441535 podStartE2EDuration="22.976733486s" totalImagesPullingTime="20.409291951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:26:41.976646813 +0000 UTC m=+23.609198798" watchObservedRunningTime="2026-07-06 15:26:41.976733486 +0000 UTC m=+23.609285469" Jul 06 15:26:41.993195 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:41.993164 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jwksx" podStartSLOduration=2.614689096 podStartE2EDuration="22.993153542s" totalImagesPullingTime="20.378464446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:26:41.992939008 +0000 UTC m=+23.625490991" watchObservedRunningTime="2026-07-06 15:26:41.993153542 +0000 UTC m=+23.625705524" Jul 06 15:26:42.010550 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:42.010497 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lhbjw" podStartSLOduration=2.691247721 podStartE2EDuration="23.010486481s" totalImagesPullingTime="20.31923876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:26:42.010036197 +0000 UTC m=+23.642588190" watchObservedRunningTime="2026-07-06 15:26:42.010486481 +0000 UTC m=+23.643038463" Jul 06 15:26:42.027592 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:42.027560 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w8l5z" podStartSLOduration=2.660379819 podStartE2EDuration="23.027520148s" totalImagesPullingTime="20.367140329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:26:42.027303562 +0000 UTC m=+23.659855546" watchObservedRunningTime="2026-07-06 15:26:42.027520148 +0000 UTC m=+23.660072131" Jul 06 15:26:42.074018 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:42.073996 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:42.143744 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:42.143713 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:42.246012 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:42.245981 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:42.797803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:42.797769 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:42.797803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:42.797796 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:42.798026 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:42.797907 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:42.798026 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:42.797966 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:42.798026 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:42.798000 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:42.798181 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:42.798098 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:42.916170 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:42.916131 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kxmff" event={"ID":"3f61903e-f088-4e01-822c-a13af1322cda","Type":"ContainerStarted","Data":"5dc634b77f4e431148e081ca9f3138a399427646f9717bc0a3502803246f5d27"} Jul 06 15:26:42.920368 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:42.919821 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" event={"ID":"87582e94-73f6-4fc5-bcd8-07fda52170b5","Type":"ContainerStarted","Data":"35fc8a046d887b1d77dd74421aac6745343c550c23bb963192d71f722573a60e"} Jul 06 15:26:42.920368 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:42.919917 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" event={"ID":"87582e94-73f6-4fc5-bcd8-07fda52170b5","Type":"ContainerStarted","Data":"93dcb5f24de2c62548bfcae33bbb2912a869cf3c306b848a069f60d5c32a4edd"} Jul 06 15:26:42.920368 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:42.919971 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" event={"ID":"87582e94-73f6-4fc5-bcd8-07fda52170b5","Type":"ContainerStarted","Data":"b5777df9aa27cd48838631b0d181fbaa5f8065afa15bead3522550eb330987f8"} Jul 06 15:26:42.920368 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:42.920021 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" event={"ID":"87582e94-73f6-4fc5-bcd8-07fda52170b5","Type":"ContainerStarted","Data":"4b81e173f532f03bddc30530067eda9546062750387ee0b834e72e01f0af0a17"} Jul 06 15:26:42.932236 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:42.932189 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kxmff" podStartSLOduration=3.55964144 podStartE2EDuration="23.932172911s" totalImagesPullingTime="20.372531471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:26:42.932083638 +0000 UTC m=+24.564635621" watchObservedRunningTime="2026-07-06 15:26:42.932172911 +0000 UTC m=+24.564724895" Jul 06 15:26:43.051352 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:43.051280 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:43.151285 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:43.151258 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:43.252544 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:43.252511 2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 15:26:43.768061 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:43.767957 2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T15:26:43.252542256Z","UUID":"28d2e9d7-fe31-42a1-b75a-e322967f25fb","Handler":null,"Name":"","Endpoint":""} Jul 06 15:26:43.769505 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:43.769483 2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 15:26:43.769677 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:43.769545 2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 15:26:43.924561 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:43.924469 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" event={"ID":"348ff35a-122a-49be-8f2a-2a1dd1032b56","Type":"ContainerStarted","Data":"6c3a703428bca7613dd6fede051940c8057e56baed01871b35dc004864aa354b"} Jul 06 15:26:43.946872 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:43.946837 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:44.006964 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:44.006874 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:44.575679 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:44.575644 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:44.633288 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:44.633247 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:44.793582 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:44.793551 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:44.793582 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:44.793567 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:44.793582 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:44.793582 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:44.793853 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:44.793675 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:44.793853 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:44.793799 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:44.793950 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:44.793889 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:44.930110 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:44.929912 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" event={"ID":"87582e94-73f6-4fc5-bcd8-07fda52170b5","Type":"ContainerStarted","Data":"4d1dc456d12ca9a0f82367b2c9b7e96caed929d89748415cec234b676aad862d"} Jul 06 15:26:44.931731 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:44.931712 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" event={"ID":"348ff35a-122a-49be-8f2a-2a1dd1032b56","Type":"ContainerStarted","Data":"95b50bd944557dda675c982ff8061d180f08dbcf96cf0f261e1fbd068cde1f78"} Jul 06 15:26:44.961722 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:44.961634 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kwvxj" podStartSLOduration=11.300474041 podStartE2EDuration="25.961612118s" totalImagesPullingTime="14.661138077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:26:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:26:44.960851473 +0000 UTC m=+26.593403457" watchObservedRunningTime="2026-07-06 15:26:44.961612118 +0000 UTC m=+26.594164102" Jul 06 15:26:46.121586 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:46.121014 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lhbjw" Jul 06 15:26:46.122260 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:46.121867 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lhbjw" Jul 06 15:26:46.794406 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:46.794372 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:46.794591 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:46.794375 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:46.794591 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:46.794521 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:46.794700 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:46.794551 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:46.794741 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:46.794705 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:46.794793 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:46.794770 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:46.936730 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:46.936647 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lhbjw" Jul 06 15:26:46.937223 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:46.937204 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lhbjw" Jul 06 15:26:47.012736 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:47.012706 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:47.942942 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:47.942744 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" event={"ID":"87582e94-73f6-4fc5-bcd8-07fda52170b5","Type":"ContainerStarted","Data":"7c9c69380886b751e7458263186871984d536afb0e66d4b308a47d1e87289975"} Jul 06 15:26:47.975780 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:47.975739 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" podStartSLOduration=8.511601167 podStartE2EDuration="28.975720547s" totalImagesPullingTime="20.46411938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:26:47.975023421 +0000 UTC m=+29.607575403" watchObservedRunningTime="2026-07-06 15:26:47.975720547 +0000 UTC m=+29.608272530" Jul 06 15:26:48.794047 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:48.794010 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:48.794254 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:48.794087 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:48.794254 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:48.794161 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:48.794334 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:48.794261 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:48.794334 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:48.794313 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:48.794402 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:48.794392 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:48.946306 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:48.946276 2561 generic.go:356] "Generic (PLEG): container finished" podID="c70229f0-9a94-4e50-84c6-6c549c7cb2db" containerID="0784eedd8d4bc18d22798771155e82fc83cbde6e64f9b90f5022f2b6e3411108" exitCode=0 Jul 06 15:26:48.946809 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:48.946332 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2h7f" event={"ID":"c70229f0-9a94-4e50-84c6-6c549c7cb2db","Type":"ContainerDied","Data":"0784eedd8d4bc18d22798771155e82fc83cbde6e64f9b90f5022f2b6e3411108"} Jul 06 15:26:48.947430 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:48.946963 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:48.947430 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:48.947033 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:48.947430 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:48.947067 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:48.964980 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:48.964953 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:48.966373 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:48.966353 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:26:49.731764 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:49.731736 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-77mbr"] Jul 06 15:26:49.731911 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:49.731831 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:49.731985 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:49.731932 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:49.733239 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:49.733217 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m4llb"] Jul 06 15:26:49.733365 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:49.733298 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:49.733426 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:49.733378 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:49.745033 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:49.745012 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9hpcm"] Jul 06 15:26:49.745122 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:49.745086 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:49.745177 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:49.745160 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:49.953503 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:49.952904 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2h7f" event={"ID":"c70229f0-9a94-4e50-84c6-6c549c7cb2db","Type":"ContainerStarted","Data":"3fb2ea3c44c8d52725b38ecc3bcb49a6e6d5c05755ff660d47ae9ab0ea8e76c9"} Jul 06 15:26:50.027390 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:50.027351 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:50.107883 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:50.107813 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:50.188912 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:50.188872 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:50.271117 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:50.271095 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:50.374722 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:50.374655 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:50.468179 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:50.468152 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:50.957277 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:50.957246 2561 generic.go:356] "Generic (PLEG): container finished" podID="c70229f0-9a94-4e50-84c6-6c549c7cb2db" containerID="3fb2ea3c44c8d52725b38ecc3bcb49a6e6d5c05755ff660d47ae9ab0ea8e76c9" exitCode=0 Jul 06 15:26:50.957696 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:50.957321 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2h7f" event={"ID":"c70229f0-9a94-4e50-84c6-6c549c7cb2db","Type":"ContainerDied","Data":"3fb2ea3c44c8d52725b38ecc3bcb49a6e6d5c05755ff660d47ae9ab0ea8e76c9"} Jul 06 15:26:51.046726 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:51.046701 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:51.146067 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:51.146038 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:51.451502 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:51.451478 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:51.527946 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:51.527912 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:51.794116 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:51.794086 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:51.794116 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:51.794115 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:51.794318 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:51.794086 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:51.794318 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:51.794193 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:51.794318 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:51.794256 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:51.794424 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:51.794347 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:52.456392 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:52.456331 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:52.456855 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:52.456452 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:26:52.456855 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:52.456500 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret podName:b4c84634-a9f3-4cc5-a67b-09805e8f67af nodeName:}" failed. No retries permitted until 2026-07-06 15:27:24.456487213 +0000 UTC m=+66.089039173 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret") pod "global-pull-secret-syncer-77mbr" (UID: "b4c84634-a9f3-4cc5-a67b-09805e8f67af") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:26:52.557610 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:52.557588 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs\") pod \"network-metrics-daemon-m4llb\" (UID: \"d047efe8-bfb0-4fee-a069-15a344dc4726\") " pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:52.557741 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:52.557721 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:26:52.557790 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:52.557768 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs podName:d047efe8-bfb0-4fee-a069-15a344dc4726 nodeName:}" failed. No retries permitted until 2026-07-06 15:27:24.557754616 +0000 UTC m=+66.190306577 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs") pod "network-metrics-daemon-m4llb" (UID: "d047efe8-bfb0-4fee-a069-15a344dc4726") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:26:52.657998 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:52.657971 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtsnt\" (UniqueName: \"kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt\") pod \"network-check-target-9hpcm\" (UID: \"30f35138-27d5-4f8c-ae95-ed63e6de45ae\") " pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:52.658120 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:52.658106 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:26:52.658164 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:52.658125 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:26:52.658164 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:52.658135 2561 projected.go:196] Error preparing data for projected volume kube-api-access-mtsnt for pod openshift-network-diagnostics/network-check-target-9hpcm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:26:52.658242 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:52.658179 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt podName:30f35138-27d5-4f8c-ae95-ed63e6de45ae nodeName:}" failed. No retries permitted until 2026-07-06 15:27:24.658163497 +0000 UTC m=+66.290715475 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mtsnt" (UniqueName: "kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt") pod "network-check-target-9hpcm" (UID: "30f35138-27d5-4f8c-ae95-ed63e6de45ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:26:52.963154 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:52.963123 2561 generic.go:356] "Generic (PLEG): container finished" podID="c70229f0-9a94-4e50-84c6-6c549c7cb2db" containerID="68cb6e9df1e25aeac61a0c043cd313d988be394e0791f922c89adbc4df1ef3e1" exitCode=0 Jul 06 15:26:52.963274 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:52.963183 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2h7f" event={"ID":"c70229f0-9a94-4e50-84c6-6c549c7cb2db","Type":"ContainerDied","Data":"68cb6e9df1e25aeac61a0c043cd313d988be394e0791f922c89adbc4df1ef3e1"} Jul 06 15:26:53.622179 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:53.622149 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:53.692848 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:53.692813 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:53.794237 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:53.794207 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:53.794237 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:53.794219 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:53.794467 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:53.794303 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:53.794467 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:53.794324 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-77mbr" podUID="b4c84634-a9f3-4cc5-a67b-09805e8f67af" Jul 06 15:26:53.794467 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:53.794395 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m4llb" podUID="d047efe8-bfb0-4fee-a069-15a344dc4726" Jul 06 15:26:53.794467 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:53.794440 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9hpcm" podUID="30f35138-27d5-4f8c-ae95-ed63e6de45ae" Jul 06 15:26:54.178781 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.178756 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-225.ec2.internal" event="NodeReady" Jul 06 15:26:54.178952 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.178889 2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 15:26:54.265442 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.265410 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6bhc8"] Jul 06 15:26:54.269052 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.269025 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6bhc8" Jul 06 15:26:54.272306 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.272287 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lrpnb"] Jul 06 15:26:54.272813 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.272789 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cwwg8\"" Jul 06 15:26:54.272897 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.272797 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 15:26:54.273866 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.273702 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 15:26:54.276473 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.276313 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lrpnb" Jul 06 15:26:54.279469 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.278984 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 15:26:54.279469 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.279050 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6q4fd\"" Jul 06 15:26:54.279469 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.279083 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 15:26:54.279469 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.279307 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 15:26:54.283937 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.283919 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6bhc8"] Jul 06 15:26:54.287433 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.287412 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lrpnb"] Jul 06 15:26:54.371069 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.371046 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-tmp-dir\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:26:54.371187 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.371100 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vnl5\" (UniqueName: \"kubernetes.io/projected/fffc4cf2-1b41-40c5-943f-f01cc52933b7-kube-api-access-8vnl5\") pod \"ingress-canary-lrpnb\" (UID: \"fffc4cf2-1b41-40c5-943f-f01cc52933b7\") " pod="openshift-ingress-canary/ingress-canary-lrpnb" Jul 06 15:26:54.371187 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.371135 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fffc4cf2-1b41-40c5-943f-f01cc52933b7-cert\") pod \"ingress-canary-lrpnb\" (UID: \"fffc4cf2-1b41-40c5-943f-f01cc52933b7\") " pod="openshift-ingress-canary/ingress-canary-lrpnb" Jul 06 15:26:54.371187 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.371155 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mm54\" (UniqueName: \"kubernetes.io/projected/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-kube-api-access-5mm54\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:26:54.371187 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.371174 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:26:54.371353 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.371205 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-config-volume\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:26:54.412287 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.412262 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:54.472135 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.472105 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-config-volume\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:26:54.472251 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.472147 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-tmp-dir\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:26:54.472251 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.472200 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vnl5\" (UniqueName: \"kubernetes.io/projected/fffc4cf2-1b41-40c5-943f-f01cc52933b7-kube-api-access-8vnl5\") pod \"ingress-canary-lrpnb\" (UID: \"fffc4cf2-1b41-40c5-943f-f01cc52933b7\") " pod="openshift-ingress-canary/ingress-canary-lrpnb" Jul 06 15:26:54.472251 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.472241 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fffc4cf2-1b41-40c5-943f-f01cc52933b7-cert\") pod \"ingress-canary-lrpnb\" (UID: \"fffc4cf2-1b41-40c5-943f-f01cc52933b7\") " pod="openshift-ingress-canary/ingress-canary-lrpnb" Jul 06 15:26:54.472411 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.472265 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mm54\" (UniqueName: \"kubernetes.io/projected/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-kube-api-access-5mm54\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:26:54.472411 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.472285 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:26:54.472513 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:54.472405 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:26:54.472513 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:54.472471 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls podName:cb50cb21-f13c-4d8c-ac8a-088425ccbd50 nodeName:}" failed. No retries permitted until 2026-07-06 15:26:54.972450304 +0000 UTC m=+36.605002282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls") pod "dns-default-6bhc8" (UID: "cb50cb21-f13c-4d8c-ac8a-088425ccbd50") : secret "dns-default-metrics-tls" not found Jul 06 15:26:54.472745 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.472717 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-tmp-dir\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:26:54.472837 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.472810 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-config-volume\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:26:54.477156 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.477132 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fffc4cf2-1b41-40c5-943f-f01cc52933b7-cert\") pod \"ingress-canary-lrpnb\" (UID: \"fffc4cf2-1b41-40c5-943f-f01cc52933b7\") " pod="openshift-ingress-canary/ingress-canary-lrpnb" Jul 06 15:26:54.483929 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.483902 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mm54\" (UniqueName: \"kubernetes.io/projected/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-kube-api-access-5mm54\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:26:54.484516 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.484493 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vnl5\" (UniqueName: \"kubernetes.io/projected/fffc4cf2-1b41-40c5-943f-f01cc52933b7-kube-api-access-8vnl5\") pod \"ingress-canary-lrpnb\" (UID: \"fffc4cf2-1b41-40c5-943f-f01cc52933b7\") " pod="openshift-ingress-canary/ingress-canary-lrpnb" Jul 06 15:26:54.515831 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.515764 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:54.589184 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.589158 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lrpnb" Jul 06 15:26:54.739567 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.739392 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lrpnb"] Jul 06 15:26:54.742904 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:26:54.742878 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfffc4cf2_1b41_40c5_943f_f01cc52933b7.slice/crio-35234d171c04a30bf9ca34dcf1f53baa6742af759fdfad491da9e6a25480181e WatchSource:0}: Error finding container 35234d171c04a30bf9ca34dcf1f53baa6742af759fdfad491da9e6a25480181e: Status 404 returned error can't find the container with id 35234d171c04a30bf9ca34dcf1f53baa6742af759fdfad491da9e6a25480181e Jul 06 15:26:54.968691 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.968584 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lrpnb" event={"ID":"fffc4cf2-1b41-40c5-943f-f01cc52933b7","Type":"ContainerStarted","Data":"35234d171c04a30bf9ca34dcf1f53baa6742af759fdfad491da9e6a25480181e"} Jul 06 15:26:54.975966 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:54.975940 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:26:54.976102 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:54.976091 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:26:54.976157 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:54.976143 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls podName:cb50cb21-f13c-4d8c-ac8a-088425ccbd50 nodeName:}" failed. No retries permitted until 2026-07-06 15:26:55.976130436 +0000 UTC m=+37.608682396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls") pod "dns-default-6bhc8" (UID: "cb50cb21-f13c-4d8c-ac8a-088425ccbd50") : secret "dns-default-metrics-tls" not found Jul 06 15:26:55.793430 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:55.793393 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:26:55.793430 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:55.793421 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:26:55.793999 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:55.793793 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:26:55.797011 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:55.796992 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 15:26:55.798470 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:55.798450 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 15:26:55.798585 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:55.798512 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5xpkj\"" Jul 06 15:26:55.798645 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:55.798599 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 15:26:55.798645 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:55.798518 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-km2tx\"" Jul 06 15:26:55.798748 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:55.798450 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 15:26:55.985422 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:55.985389 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:26:55.985605 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:55.985554 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:26:55.985694 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:55.985607 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls podName:cb50cb21-f13c-4d8c-ac8a-088425ccbd50 nodeName:}" failed. No retries permitted until 2026-07-06 15:26:57.98558987 +0000 UTC m=+39.618141831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls") pod "dns-default-6bhc8" (UID: "cb50cb21-f13c-4d8c-ac8a-088425ccbd50") : secret "dns-default-metrics-tls" not found Jul 06 15:26:58.000937 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:58.000897 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:26:58.001375 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:58.001064 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:26:58.001375 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:26:58.001145 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls podName:cb50cb21-f13c-4d8c-ac8a-088425ccbd50 nodeName:}" failed. No retries permitted until 2026-07-06 15:27:02.001124186 +0000 UTC m=+43.633676165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls") pod "dns-default-6bhc8" (UID: "cb50cb21-f13c-4d8c-ac8a-088425ccbd50") : secret "dns-default-metrics-tls" not found Jul 06 15:26:59.015004 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:59.014976 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:59.105774 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:59.105692 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:26:59.984831 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:59.984797 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lrpnb" event={"ID":"fffc4cf2-1b41-40c5-943f-f01cc52933b7","Type":"ContainerStarted","Data":"c1b41cdfa03ac676825a91864a2f39278ac2bf3f74d977919d975b9d663286ef"} Jul 06 15:26:59.987458 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:59.987434 2561 generic.go:356] "Generic (PLEG): container finished" podID="c70229f0-9a94-4e50-84c6-6c549c7cb2db" containerID="d04ec7162080e2aacd7c35dbeb4522fe68757d8442a48dba63c3a12e132b8c1f" exitCode=0 Jul 06 15:26:59.987591 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:26:59.987470 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2h7f" event={"ID":"c70229f0-9a94-4e50-84c6-6c549c7cb2db","Type":"ContainerDied","Data":"d04ec7162080e2aacd7c35dbeb4522fe68757d8442a48dba63c3a12e132b8c1f"} Jul 06 15:27:00.003976 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:00.003940 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lrpnb" podStartSLOduration=1.813617958 podStartE2EDuration="6.003928206s" totalImagesPullingTime="4.190310248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:27:00.003661662 +0000 UTC m=+41.636213638" watchObservedRunningTime="2026-07-06 15:27:00.003928206 +0000 UTC m=+41.636480207" Jul 06 15:27:00.992934 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:00.992901 2561 generic.go:356] "Generic (PLEG): container finished" podID="c70229f0-9a94-4e50-84c6-6c549c7cb2db" containerID="cc4651dddd9e419383ff465505108e20b11c71cc10d20e65e6ab945209d99efc" exitCode=0 Jul 06 15:27:00.993485 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:00.992992 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2h7f" event={"ID":"c70229f0-9a94-4e50-84c6-6c549c7cb2db","Type":"ContainerDied","Data":"cc4651dddd9e419383ff465505108e20b11c71cc10d20e65e6ab945209d99efc"} Jul 06 15:27:01.998711 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:01.998683 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m2h7f" event={"ID":"c70229f0-9a94-4e50-84c6-6c549c7cb2db","Type":"ContainerStarted","Data":"31447fc2d494010d6ea8e836a40a87132634aa28c8d685b219a8bfc96f5254ed"} Jul 06 15:27:02.032188 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:02.029934 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:27:02.032326 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:27:02.032226 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:27:02.032577 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:27:02.032559 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls podName:cb50cb21-f13c-4d8c-ac8a-088425ccbd50 nodeName:}" failed. No retries permitted until 2026-07-06 15:27:10.03252209 +0000 UTC m=+51.665074050 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls") pod "dns-default-6bhc8" (UID: "cb50cb21-f13c-4d8c-ac8a-088425ccbd50") : secret "dns-default-metrics-tls" not found Jul 06 15:27:02.040262 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:02.040224 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m2h7f" podStartSLOduration=8.60474093 podStartE2EDuration="43.040213244s" totalImagesPullingTime="34.435472314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:26:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:27:02.040030903 +0000 UTC m=+43.672582901" watchObservedRunningTime="2026-07-06 15:27:02.040213244 +0000 UTC m=+43.672765226" Jul 06 15:27:02.192182 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:02.192151 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:02.263240 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:02.263177 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:05.583338 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:05.583306 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:05.683651 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:05.683618 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:10.082907 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:10.082873 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:27:10.083260 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:27:10.083017 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:27:10.083260 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:27:10.083085 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls podName:cb50cb21-f13c-4d8c-ac8a-088425ccbd50 nodeName:}" failed. No retries permitted until 2026-07-06 15:27:26.083066747 +0000 UTC m=+67.715618713 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls") pod "dns-default-6bhc8" (UID: "cb50cb21-f13c-4d8c-ac8a-088425ccbd50") : secret "dns-default-metrics-tls" not found Jul 06 15:27:12.480262 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:12.480222 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:12.577673 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:12.577648 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:13.423510 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:13.423474 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:13.523486 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:13.523456 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:14.348340 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:14.348306 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:20.974198 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:20.974171 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-562c5" Jul 06 15:27:21.157934 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:21.157908 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:21.263864 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:21.263786 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:24.481702 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:24.481663 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:27:24.485059 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:24.485041 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 15:27:24.494974 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:24.494944 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4c84634-a9f3-4cc5-a67b-09805e8f67af-original-pull-secret\") pod \"global-pull-secret-syncer-77mbr\" (UID: \"b4c84634-a9f3-4cc5-a67b-09805e8f67af\") " pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:27:24.582276 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:24.582251 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs\") pod \"network-metrics-daemon-m4llb\" (UID: \"d047efe8-bfb0-4fee-a069-15a344dc4726\") " pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:27:24.585298 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:24.585281 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 15:27:24.592636 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:27:24.592619 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 06 15:27:24.592693 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:27:24.592682 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs podName:d047efe8-bfb0-4fee-a069-15a344dc4726 nodeName:}" failed. No retries permitted until 2026-07-06 15:28:28.592666177 +0000 UTC m=+130.225218137 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs") pod "network-metrics-daemon-m4llb" (UID: "d047efe8-bfb0-4fee-a069-15a344dc4726") : secret "metrics-daemon-secret" not found Jul 06 15:27:24.617136 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:24.617113 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-77mbr" Jul 06 15:27:24.683423 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:24.683392 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtsnt\" (UniqueName: \"kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt\") pod \"network-check-target-9hpcm\" (UID: \"30f35138-27d5-4f8c-ae95-ed63e6de45ae\") " pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:27:24.686674 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:24.686650 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 15:27:24.696722 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:24.696699 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 15:27:24.707287 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:24.707260 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtsnt\" (UniqueName: \"kubernetes.io/projected/30f35138-27d5-4f8c-ae95-ed63e6de45ae-kube-api-access-mtsnt\") pod \"network-check-target-9hpcm\" (UID: \"30f35138-27d5-4f8c-ae95-ed63e6de45ae\") " pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:27:24.749070 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:24.749006 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-77mbr"] Jul 06 15:27:24.752882 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:27:24.752857 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c84634_a9f3_4cc5_a67b_09805e8f67af.slice/crio-eb72df2ba6a1d087124bfbb682453598dab494341827b55e611d214304e00bb3 WatchSource:0}: Error finding container eb72df2ba6a1d087124bfbb682453598dab494341827b55e611d214304e00bb3: Status 404 returned error can't find the container with id eb72df2ba6a1d087124bfbb682453598dab494341827b55e611d214304e00bb3 Jul 06 15:27:24.782431 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:24.782406 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:24.875672 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:24.875649 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:24.910701 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:24.910677 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5xpkj\"" Jul 06 15:27:24.918139 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:24.918122 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:27:25.049807 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:25.049779 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-77mbr" event={"ID":"b4c84634-a9f3-4cc5-a67b-09805e8f67af","Type":"ContainerStarted","Data":"eb72df2ba6a1d087124bfbb682453598dab494341827b55e611d214304e00bb3"} Jul 06 15:27:25.060149 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:25.060110 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9hpcm"] Jul 06 15:27:25.061979 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:27:25.061955 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f35138_27d5_4f8c_ae95_ed63e6de45ae.slice/crio-3efd1e1c14f81b6c332334ecb6f94383c9ca5e4de17dd1019b2d64c80b0d21df WatchSource:0}: Error finding container 3efd1e1c14f81b6c332334ecb6f94383c9ca5e4de17dd1019b2d64c80b0d21df: Status 404 returned error can't find the container with id 3efd1e1c14f81b6c332334ecb6f94383c9ca5e4de17dd1019b2d64c80b0d21df Jul 06 15:27:26.053875 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:26.053831 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9hpcm" event={"ID":"30f35138-27d5-4f8c-ae95-ed63e6de45ae","Type":"ContainerStarted","Data":"3efd1e1c14f81b6c332334ecb6f94383c9ca5e4de17dd1019b2d64c80b0d21df"} Jul 06 15:27:26.093517 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:26.093484 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:27:26.093680 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:27:26.093631 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:27:26.093725 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:27:26.093688 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls podName:cb50cb21-f13c-4d8c-ac8a-088425ccbd50 nodeName:}" failed. No retries permitted until 2026-07-06 15:27:58.093672082 +0000 UTC m=+99.726224046 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls") pod "dns-default-6bhc8" (UID: "cb50cb21-f13c-4d8c-ac8a-088425ccbd50") : secret "dns-default-metrics-tls" not found Jul 06 15:27:28.700087 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:28.700034 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:30.050461 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:30.050430 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:30.065364 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:30.065331 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-77mbr" event={"ID":"b4c84634-a9f3-4cc5-a67b-09805e8f67af","Type":"ContainerStarted","Data":"755852d2ba69adc25e112c28346e122b667ef5102334f6f2fa8ae40bbc81e837"} Jul 06 15:27:30.067005 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:30.066984 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9hpcm" event={"ID":"30f35138-27d5-4f8c-ae95-ed63e6de45ae","Type":"ContainerStarted","Data":"77cd7cea16c0f2c8a6ff27c8d56399233549852a38b1138a0589f40eb26138d6"} Jul 06 15:27:30.067124 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:30.067109 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:27:30.084767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:30.084729 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-77mbr" podStartSLOduration=66.435654285 podStartE2EDuration="1m11.084715647s" totalImagesPullingTime="4.649061362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:27:30.084016274 +0000 UTC m=+71.716568258" watchObservedRunningTime="2026-07-06 15:27:30.084715647 +0000 UTC m=+71.717267630" Jul 06 15:27:50.469043 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:50.468921 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:50.548813 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:50.548758 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:54.891867 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:54.891836 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:54.992313 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:54.992277 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:57.835303 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:57.835268 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:27:58.100747 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:27:58.100658 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:27:58.100883 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:27:58.100769 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:27:58.100883 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:27:58.100829 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls podName:cb50cb21-f13c-4d8c-ac8a-088425ccbd50 nodeName:}" failed. No retries permitted until 2026-07-06 15:29:02.100816638 +0000 UTC m=+163.733368599 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls") pod "dns-default-6bhc8" (UID: "cb50cb21-f13c-4d8c-ac8a-088425ccbd50") : secret "dns-default-metrics-tls" not found Jul 06 15:28:01.072248 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:01.072219 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9hpcm" Jul 06 15:28:01.091382 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:01.091331 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9hpcm" podStartSLOduration=97.756319283 podStartE2EDuration="1m42.091317172s" totalImagesPullingTime="4.334997889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:27:30.101022235 +0000 UTC m=+71.733574219" watchObservedRunningTime="2026-07-06 15:28:01.091317172 +0000 UTC m=+102.723869147" Jul 06 15:28:04.923437 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:04.923404 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-862fp"] Jul 06 15:28:04.926177 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:04.926161 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-862fp" Jul 06 15:28:04.929214 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:04.929193 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ghwjb\"" Jul 06 15:28:04.937228 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:04.937205 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-862fp"] Jul 06 15:28:05.026630 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.026603 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn"] Jul 06 15:28:05.028453 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.028438 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" Jul 06 15:28:05.031250 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.031229 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 06 15:28:05.031409 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.031390 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 06 15:28:05.031719 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.031701 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gjcvt\"" Jul 06 15:28:05.031810 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.031781 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:28:05.039650 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.039632 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9lqk\" (UniqueName: \"kubernetes.io/projected/1a3e3254-bbc1-4315-a976-ae3821cc6b84-kube-api-access-t9lqk\") pod \"network-check-source-989f8b8dc-862fp\" (UID: \"1a3e3254-bbc1-4315-a976-ae3821cc6b84\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-862fp" Jul 06 15:28:05.041434 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.041413 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn"] Jul 06 15:28:05.127629 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.127600 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g"] Jul 06 15:28:05.129343 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.129329 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" Jul 06 15:28:05.131851 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.131831 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-xmrmx"] Jul 06 15:28:05.133555 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.133523 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-q2lgf"] Jul 06 15:28:05.133632 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.133608 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.135438 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.135421 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:05.135791 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.135771 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8cz5c\"" Jul 06 15:28:05.135888 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.135816 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 06 15:28:05.135934 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.135901 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 06 15:28:05.135978 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.135947 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 06 15:28:05.135978 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.135953 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:28:05.140114 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.140092 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9lqk\" (UniqueName: \"kubernetes.io/projected/1a3e3254-bbc1-4315-a976-ae3821cc6b84-kube-api-access-t9lqk\") pod \"network-check-source-989f8b8dc-862fp\" (UID: \"1a3e3254-bbc1-4315-a976-ae3821cc6b84\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-862fp" Jul 06 15:28:05.140219 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.140144 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jmbm\" (UniqueName: \"kubernetes.io/projected/17fdbe64-b3a3-42b9-91eb-68b74152fdda-kube-api-access-6jmbm\") pod \"cluster-samples-operator-56fcf86bf-ls2rn\" (UID: \"17fdbe64-b3a3-42b9-91eb-68b74152fdda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" Jul 06 15:28:05.140294 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.140279 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ls2rn\" (UID: \"17fdbe64-b3a3-42b9-91eb-68b74152fdda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" Jul 06 15:28:05.145999 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.145977 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 15:28:05.147743 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.147722 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ds7wd\"" Jul 06 15:28:05.148015 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.147994 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 15:28:05.149412 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.148631 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 15:28:05.149412 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.148745 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 15:28:05.149412 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.148865 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:28:05.149412 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.148970 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 06 15:28:05.149412 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.149052 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 06 15:28:05.149412 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.149098 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-456qr\"" Jul 06 15:28:05.149412 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.149168 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 06 15:28:05.150465 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.150375 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g"] Jul 06 15:28:05.151485 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.151465 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-xmrmx"] Jul 06 15:28:05.151724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.151701 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 15:28:05.151823 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.151773 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 06 15:28:05.152352 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.152307 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-q2lgf"] Jul 06 15:28:05.156289 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.156230 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9lqk\" (UniqueName: \"kubernetes.io/projected/1a3e3254-bbc1-4315-a976-ae3821cc6b84-kube-api-access-t9lqk\") pod \"network-check-source-989f8b8dc-862fp\" (UID: \"1a3e3254-bbc1-4315-a976-ae3821cc6b84\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-862fp" Jul 06 15:28:05.208207 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.208144 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:05.227078 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.227053 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj"] Jul 06 15:28:05.228977 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.228960 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" Jul 06 15:28:05.231790 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.231768 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 06 15:28:05.231888 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.231793 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tckqm\"" Jul 06 15:28:05.231888 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.231811 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 06 15:28:05.231888 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.231819 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 06 15:28:05.232018 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.231943 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:28:05.233941 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.233924 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-862fp" Jul 06 15:28:05.240681 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.240661 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/48ef32e0-be01-49ac-ae54-32f33476eadc-serving-cert\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.240777 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.240694 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6283d26a-0676-4b9e-be5b-7a41a44079ae-config\") pod \"console-operator-68b757fb67-q2lgf\" (UID: \"6283d26a-0676-4b9e-be5b-7a41a44079ae\") " pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:05.240777 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.240721 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8558\" (UniqueName: \"kubernetes.io/projected/48ef32e0-be01-49ac-ae54-32f33476eadc-kube-api-access-p8558\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.240777 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.240749 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6283d26a-0676-4b9e-be5b-7a41a44079ae-trusted-ca\") pod \"console-operator-68b757fb67-q2lgf\" (UID: \"6283d26a-0676-4b9e-be5b-7a41a44079ae\") " pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:05.240905 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.240797 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48ef32e0-be01-49ac-ae54-32f33476eadc-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.240905 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.240835 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmbm\" (UniqueName: \"kubernetes.io/projected/17fdbe64-b3a3-42b9-91eb-68b74152fdda-kube-api-access-6jmbm\") pod \"cluster-samples-operator-56fcf86bf-ls2rn\" (UID: \"17fdbe64-b3a3-42b9-91eb-68b74152fdda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" Jul 06 15:28:05.240905 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.240864 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48ef32e0-be01-49ac-ae54-32f33476eadc-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.240905 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.240894 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qtwx\" (UniqueName: \"kubernetes.io/projected/6283d26a-0676-4b9e-be5b-7a41a44079ae-kube-api-access-7qtwx\") pod \"console-operator-68b757fb67-q2lgf\" (UID: \"6283d26a-0676-4b9e-be5b-7a41a44079ae\") " pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:05.241096 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.240946 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ls2rn\" (UID: \"17fdbe64-b3a3-42b9-91eb-68b74152fdda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" Jul 06 15:28:05.241096 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.240986 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6283d26a-0676-4b9e-be5b-7a41a44079ae-serving-cert\") pod \"console-operator-68b757fb67-q2lgf\" (UID: \"6283d26a-0676-4b9e-be5b-7a41a44079ae\") " pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:05.241096 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.241024 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7svf\" (UniqueName: \"kubernetes.io/projected/705cc873-6a42-42ad-8bdb-dbc28545a42f-kube-api-access-n7svf\") pod \"service-ca-operator-5d46d9ffd4-f5b2g\" (UID: \"705cc873-6a42-42ad-8bdb-dbc28545a42f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" Jul 06 15:28:05.241096 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:05.241038 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 15:28:05.241096 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.241057 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48ef32e0-be01-49ac-ae54-32f33476eadc-tmp\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.241096 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:05.241087 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls podName:17fdbe64-b3a3-42b9-91eb-68b74152fdda nodeName:}" failed. No retries permitted until 2026-07-06 15:28:05.741070624 +0000 UTC m=+107.373622603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ls2rn" (UID: "17fdbe64-b3a3-42b9-91eb-68b74152fdda") : secret "samples-operator-tls" not found Jul 06 15:28:05.241411 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.241121 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/48ef32e0-be01-49ac-ae54-32f33476eadc-snapshots\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.241411 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.241150 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/705cc873-6a42-42ad-8bdb-dbc28545a42f-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-f5b2g\" (UID: \"705cc873-6a42-42ad-8bdb-dbc28545a42f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" Jul 06 15:28:05.241411 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.241213 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/705cc873-6a42-42ad-8bdb-dbc28545a42f-config\") pod \"service-ca-operator-5d46d9ffd4-f5b2g\" (UID: \"705cc873-6a42-42ad-8bdb-dbc28545a42f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" Jul 06 15:28:05.241819 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.241800 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj"] Jul 06 15:28:05.255405 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.255384 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmbm\" (UniqueName: \"kubernetes.io/projected/17fdbe64-b3a3-42b9-91eb-68b74152fdda-kube-api-access-6jmbm\") pod \"cluster-samples-operator-56fcf86bf-ls2rn\" (UID: \"17fdbe64-b3a3-42b9-91eb-68b74152fdda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" Jul 06 15:28:05.287308 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.287253 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:05.342178 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342147 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/48ef32e0-be01-49ac-ae54-32f33476eadc-snapshots\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.342314 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342199 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/705cc873-6a42-42ad-8bdb-dbc28545a42f-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-f5b2g\" (UID: \"705cc873-6a42-42ad-8bdb-dbc28545a42f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" Jul 06 15:28:05.342314 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342232 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/705cc873-6a42-42ad-8bdb-dbc28545a42f-config\") pod \"service-ca-operator-5d46d9ffd4-f5b2g\" (UID: \"705cc873-6a42-42ad-8bdb-dbc28545a42f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" Jul 06 15:28:05.342314 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342254 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6dbf566-cef0-4ead-a469-1a197bce5b3d-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-px9pj\" (UID: \"f6dbf566-cef0-4ead-a469-1a197bce5b3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" Jul 06 15:28:05.342314 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342274 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdmnx\" (UniqueName: \"kubernetes.io/projected/f6dbf566-cef0-4ead-a469-1a197bce5b3d-kube-api-access-cdmnx\") pod \"kube-storage-version-migrator-operator-846bf668f7-px9pj\" (UID: \"f6dbf566-cef0-4ead-a469-1a197bce5b3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" Jul 06 15:28:05.342314 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342305 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/48ef32e0-be01-49ac-ae54-32f33476eadc-serving-cert\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.342604 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342329 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6283d26a-0676-4b9e-be5b-7a41a44079ae-config\") pod \"console-operator-68b757fb67-q2lgf\" (UID: \"6283d26a-0676-4b9e-be5b-7a41a44079ae\") " pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:05.342604 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342348 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8558\" (UniqueName: \"kubernetes.io/projected/48ef32e0-be01-49ac-ae54-32f33476eadc-kube-api-access-p8558\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.342604 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342370 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6283d26a-0676-4b9e-be5b-7a41a44079ae-trusted-ca\") pod \"console-operator-68b757fb67-q2lgf\" (UID: \"6283d26a-0676-4b9e-be5b-7a41a44079ae\") " pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:05.342604 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342392 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48ef32e0-be01-49ac-ae54-32f33476eadc-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.342604 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342426 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48ef32e0-be01-49ac-ae54-32f33476eadc-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.342604 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342454 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6dbf566-cef0-4ead-a469-1a197bce5b3d-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-px9pj\" (UID: \"f6dbf566-cef0-4ead-a469-1a197bce5b3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" Jul 06 15:28:05.342604 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342480 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qtwx\" (UniqueName: \"kubernetes.io/projected/6283d26a-0676-4b9e-be5b-7a41a44079ae-kube-api-access-7qtwx\") pod \"console-operator-68b757fb67-q2lgf\" (UID: \"6283d26a-0676-4b9e-be5b-7a41a44079ae\") " pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:05.342604 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342580 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6283d26a-0676-4b9e-be5b-7a41a44079ae-serving-cert\") pod \"console-operator-68b757fb67-q2lgf\" (UID: \"6283d26a-0676-4b9e-be5b-7a41a44079ae\") " pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:05.343034 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342611 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7svf\" (UniqueName: \"kubernetes.io/projected/705cc873-6a42-42ad-8bdb-dbc28545a42f-kube-api-access-n7svf\") pod \"service-ca-operator-5d46d9ffd4-f5b2g\" (UID: \"705cc873-6a42-42ad-8bdb-dbc28545a42f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" Jul 06 15:28:05.343034 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342643 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48ef32e0-be01-49ac-ae54-32f33476eadc-tmp\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.343034 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.342973 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/48ef32e0-be01-49ac-ae54-32f33476eadc-snapshots\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.343307 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.343082 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/705cc873-6a42-42ad-8bdb-dbc28545a42f-config\") pod \"service-ca-operator-5d46d9ffd4-f5b2g\" (UID: \"705cc873-6a42-42ad-8bdb-dbc28545a42f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" Jul 06 15:28:05.343307 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.343131 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48ef32e0-be01-49ac-ae54-32f33476eadc-tmp\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.343307 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.343152 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6283d26a-0676-4b9e-be5b-7a41a44079ae-config\") pod \"console-operator-68b757fb67-q2lgf\" (UID: \"6283d26a-0676-4b9e-be5b-7a41a44079ae\") " pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:05.343757 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.343731 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48ef32e0-be01-49ac-ae54-32f33476eadc-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.343867 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.343796 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6283d26a-0676-4b9e-be5b-7a41a44079ae-trusted-ca\") pod \"console-operator-68b757fb67-q2lgf\" (UID: \"6283d26a-0676-4b9e-be5b-7a41a44079ae\") " pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:05.344507 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.344484 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48ef32e0-be01-49ac-ae54-32f33476eadc-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.344983 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.344957 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/48ef32e0-be01-49ac-ae54-32f33476eadc-serving-cert\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.345052 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.344983 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/705cc873-6a42-42ad-8bdb-dbc28545a42f-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-f5b2g\" (UID: \"705cc873-6a42-42ad-8bdb-dbc28545a42f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" Jul 06 15:28:05.345485 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.345465 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6283d26a-0676-4b9e-be5b-7a41a44079ae-serving-cert\") pod \"console-operator-68b757fb67-q2lgf\" (UID: \"6283d26a-0676-4b9e-be5b-7a41a44079ae\") " pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:05.351644 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.351623 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7svf\" (UniqueName: \"kubernetes.io/projected/705cc873-6a42-42ad-8bdb-dbc28545a42f-kube-api-access-n7svf\") pod \"service-ca-operator-5d46d9ffd4-f5b2g\" (UID: \"705cc873-6a42-42ad-8bdb-dbc28545a42f\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" Jul 06 15:28:05.351766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.351745 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qtwx\" (UniqueName: \"kubernetes.io/projected/6283d26a-0676-4b9e-be5b-7a41a44079ae-kube-api-access-7qtwx\") pod \"console-operator-68b757fb67-q2lgf\" (UID: \"6283d26a-0676-4b9e-be5b-7a41a44079ae\") " pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:05.352571 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.352552 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8558\" (UniqueName: \"kubernetes.io/projected/48ef32e0-be01-49ac-ae54-32f33476eadc-kube-api-access-p8558\") pod \"insights-operator-5c847dcfc4-xmrmx\" (UID: \"48ef32e0-be01-49ac-ae54-32f33476eadc\") " pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.365289 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.365271 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-862fp"] Jul 06 15:28:05.367476 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:05.367455 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3e3254_bbc1_4315_a976_ae3821cc6b84.slice/crio-052e81c1d5a018ee67fc8ca76df8431eb103d490e7243d6f13c8e2318ce12f6b WatchSource:0}: Error finding container 052e81c1d5a018ee67fc8ca76df8431eb103d490e7243d6f13c8e2318ce12f6b: Status 404 returned error can't find the container with id 052e81c1d5a018ee67fc8ca76df8431eb103d490e7243d6f13c8e2318ce12f6b Jul 06 15:28:05.440399 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.440368 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" Jul 06 15:28:05.443293 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.443269 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6dbf566-cef0-4ead-a469-1a197bce5b3d-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-px9pj\" (UID: \"f6dbf566-cef0-4ead-a469-1a197bce5b3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" Jul 06 15:28:05.443392 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.443295 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdmnx\" (UniqueName: \"kubernetes.io/projected/f6dbf566-cef0-4ead-a469-1a197bce5b3d-kube-api-access-cdmnx\") pod \"kube-storage-version-migrator-operator-846bf668f7-px9pj\" (UID: \"f6dbf566-cef0-4ead-a469-1a197bce5b3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" Jul 06 15:28:05.443392 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.443331 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6dbf566-cef0-4ead-a469-1a197bce5b3d-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-px9pj\" (UID: \"f6dbf566-cef0-4ead-a469-1a197bce5b3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" Jul 06 15:28:05.443832 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.443810 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f6dbf566-cef0-4ead-a469-1a197bce5b3d-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-px9pj\" (UID: \"f6dbf566-cef0-4ead-a469-1a197bce5b3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" Jul 06 15:28:05.445936 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.445865 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f6dbf566-cef0-4ead-a469-1a197bce5b3d-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-px9pj\" (UID: \"f6dbf566-cef0-4ead-a469-1a197bce5b3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" Jul 06 15:28:05.448813 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.448791 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" Jul 06 15:28:05.456551 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.456517 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:05.456900 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.456879 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdmnx\" (UniqueName: \"kubernetes.io/projected/f6dbf566-cef0-4ead-a469-1a197bce5b3d-kube-api-access-cdmnx\") pod \"kube-storage-version-migrator-operator-846bf668f7-px9pj\" (UID: \"f6dbf566-cef0-4ead-a469-1a197bce5b3d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" Jul 06 15:28:05.537264 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.537156 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" Jul 06 15:28:05.597879 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.596938 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g"] Jul 06 15:28:05.600669 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:05.600639 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705cc873_6a42_42ad_8bdb_dbc28545a42f.slice/crio-9e1447f392207f68d5070f09993f178e1fe2cf0996d6601d23497a204a702fe3 WatchSource:0}: Error finding container 9e1447f392207f68d5070f09993f178e1fe2cf0996d6601d23497a204a702fe3: Status 404 returned error can't find the container with id 9e1447f392207f68d5070f09993f178e1fe2cf0996d6601d23497a204a702fe3 Jul 06 15:28:05.611427 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.611350 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-xmrmx"] Jul 06 15:28:05.615227 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:05.615187 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ef32e0_be01_49ac_ae54_32f33476eadc.slice/crio-eb273307b51422417902f4f3e910c8a3df131614db40c33cc4627c5ea0b1abd8 WatchSource:0}: Error finding container eb273307b51422417902f4f3e910c8a3df131614db40c33cc4627c5ea0b1abd8: Status 404 returned error can't find the container with id eb273307b51422417902f4f3e910c8a3df131614db40c33cc4627c5ea0b1abd8 Jul 06 15:28:05.635434 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.635408 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-q2lgf"] Jul 06 15:28:05.639611 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:05.639554 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6283d26a_0676_4b9e_be5b_7a41a44079ae.slice/crio-36cf2c4288803739faaa86bcf5bbe372b08ff7bf94cf340d8d69ea063a45b0a8 WatchSource:0}: Error finding container 36cf2c4288803739faaa86bcf5bbe372b08ff7bf94cf340d8d69ea063a45b0a8: Status 404 returned error can't find the container with id 36cf2c4288803739faaa86bcf5bbe372b08ff7bf94cf340d8d69ea063a45b0a8 Jul 06 15:28:05.687988 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.687961 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj"] Jul 06 15:28:05.691638 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:05.691617 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dbf566_cef0_4ead_a469_1a197bce5b3d.slice/crio-12682f3a44fdfbd651dcc7ca11621aaa2dc08cf78f59844051fa7563d5634459 WatchSource:0}: Error finding container 12682f3a44fdfbd651dcc7ca11621aaa2dc08cf78f59844051fa7563d5634459: Status 404 returned error can't find the container with id 12682f3a44fdfbd651dcc7ca11621aaa2dc08cf78f59844051fa7563d5634459 Jul 06 15:28:05.746797 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:05.746778 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ls2rn\" (UID: \"17fdbe64-b3a3-42b9-91eb-68b74152fdda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" Jul 06 15:28:05.746903 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:05.746889 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 15:28:05.746952 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:05.746943 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls podName:17fdbe64-b3a3-42b9-91eb-68b74152fdda nodeName:}" failed. No retries permitted until 2026-07-06 15:28:06.746930848 +0000 UTC m=+108.379482810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ls2rn" (UID: "17fdbe64-b3a3-42b9-91eb-68b74152fdda") : secret "samples-operator-tls" not found Jul 06 15:28:06.162103 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:06.162021 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" event={"ID":"f6dbf566-cef0-4ead-a469-1a197bce5b3d","Type":"ContainerStarted","Data":"12682f3a44fdfbd651dcc7ca11621aaa2dc08cf78f59844051fa7563d5634459"} Jul 06 15:28:06.167393 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:06.167361 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" event={"ID":"6283d26a-0676-4b9e-be5b-7a41a44079ae","Type":"ContainerStarted","Data":"36cf2c4288803739faaa86bcf5bbe372b08ff7bf94cf340d8d69ea063a45b0a8"} Jul 06 15:28:06.169140 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:06.169118 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" event={"ID":"705cc873-6a42-42ad-8bdb-dbc28545a42f","Type":"ContainerStarted","Data":"9e1447f392207f68d5070f09993f178e1fe2cf0996d6601d23497a204a702fe3"} Jul 06 15:28:06.170242 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:06.170221 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" event={"ID":"48ef32e0-be01-49ac-ae54-32f33476eadc","Type":"ContainerStarted","Data":"eb273307b51422417902f4f3e910c8a3df131614db40c33cc4627c5ea0b1abd8"} Jul 06 15:28:06.172436 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:06.172416 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-862fp" event={"ID":"1a3e3254-bbc1-4315-a976-ae3821cc6b84","Type":"ContainerStarted","Data":"921d2486850d2f439da515b26a9b73ead600630bfd717f4dbd7ef3ad1f913a82"} Jul 06 15:28:06.172561 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:06.172457 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-862fp" event={"ID":"1a3e3254-bbc1-4315-a976-ae3821cc6b84","Type":"ContainerStarted","Data":"052e81c1d5a018ee67fc8ca76df8431eb103d490e7243d6f13c8e2318ce12f6b"} Jul 06 15:28:06.755172 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:06.755135 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ls2rn\" (UID: \"17fdbe64-b3a3-42b9-91eb-68b74152fdda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" Jul 06 15:28:06.755421 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:06.755322 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 15:28:06.755421 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:06.755409 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls podName:17fdbe64-b3a3-42b9-91eb-68b74152fdda nodeName:}" failed. No retries permitted until 2026-07-06 15:28:08.755388443 +0000 UTC m=+110.387940409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ls2rn" (UID: "17fdbe64-b3a3-42b9-91eb-68b74152fdda") : secret "samples-operator-tls" not found Jul 06 15:28:07.535571 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:07.535516 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:07.625656 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:07.625619 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:08.772114 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:08.772082 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ls2rn\" (UID: \"17fdbe64-b3a3-42b9-91eb-68b74152fdda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" Jul 06 15:28:08.772564 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:08.772254 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 15:28:08.772564 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:08.772353 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls podName:17fdbe64-b3a3-42b9-91eb-68b74152fdda nodeName:}" failed. No retries permitted until 2026-07-06 15:28:12.772337279 +0000 UTC m=+114.404889239 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ls2rn" (UID: "17fdbe64-b3a3-42b9-91eb-68b74152fdda") : secret "samples-operator-tls" not found Jul 06 15:28:10.189838 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:10.189808 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/0.log" Jul 06 15:28:10.190365 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:10.189849 2561 generic.go:356] "Generic (PLEG): container finished" podID="6283d26a-0676-4b9e-be5b-7a41a44079ae" containerID="287d673977c437b6c927989050fd523962356e4ef2f89d8c7233bf392b6dc8cc" exitCode=255 Jul 06 15:28:10.190365 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:10.189915 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" event={"ID":"6283d26a-0676-4b9e-be5b-7a41a44079ae","Type":"ContainerDied","Data":"287d673977c437b6c927989050fd523962356e4ef2f89d8c7233bf392b6dc8cc"} Jul 06 15:28:10.190365 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:10.190174 2561 scope.go:117] "RemoveContainer" containerID="287d673977c437b6c927989050fd523962356e4ef2f89d8c7233bf392b6dc8cc" Jul 06 15:28:10.191844 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:10.191810 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" event={"ID":"705cc873-6a42-42ad-8bdb-dbc28545a42f","Type":"ContainerStarted","Data":"5d00d316ad5ad75ff1daa6c419b33b80fb147a4a6a4f546a523de14bdefc7b60"} Jul 06 15:28:10.193651 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:10.193480 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" event={"ID":"48ef32e0-be01-49ac-ae54-32f33476eadc","Type":"ContainerStarted","Data":"5b8d83fd45bced1b8b97cf539ec7429cf69ddcc3a0ea79bc9886e5a3b2941a38"} Jul 06 15:28:10.195097 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:10.195075 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" event={"ID":"f6dbf566-cef0-4ead-a469-1a197bce5b3d","Type":"ContainerStarted","Data":"416f5ac8b2f88f2a7888ef55f2a5f625c116498ee040390789ed4c7c11b85428"} Jul 06 15:28:10.214728 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:10.214677 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-862fp" podStartSLOduration=6.21466014 podStartE2EDuration="6.21466014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:06.19489244 +0000 UTC m=+107.827444424" watchObservedRunningTime="2026-07-06 15:28:10.21466014 +0000 UTC m=+111.847212124" Jul 06 15:28:10.233510 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:10.233463 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" podStartSLOduration=1.595841485 podStartE2EDuration="5.233447116s" totalImagesPullingTime="3.637605631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:10.23258658 +0000 UTC m=+111.865138565" watchObservedRunningTime="2026-07-06 15:28:10.233447116 +0000 UTC m=+111.865999100" Jul 06 15:28:10.252849 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:10.252804 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" podStartSLOduration=1.626064301 podStartE2EDuration="5.252786784s" totalImagesPullingTime="3.626722483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:10.252450297 +0000 UTC m=+111.885002283" watchObservedRunningTime="2026-07-06 15:28:10.252786784 +0000 UTC m=+111.885338835" Jul 06 15:28:10.272024 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:10.271965 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" podStartSLOduration=1.7177474099999999 podStartE2EDuration="5.271951577s" totalImagesPullingTime="3.554204167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:10.271590218 +0000 UTC m=+111.904142201" watchObservedRunningTime="2026-07-06 15:28:10.271951577 +0000 UTC m=+111.904503560" Jul 06 15:28:10.462892 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:10.462824 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:10.563289 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:10.563261 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:10.723150 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:10.722574 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:10.840916 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:10.840874 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:11.017115 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.017020 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:11.120970 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.120922 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:11.133390 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.133362 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9s9qg"] Jul 06 15:28:11.136646 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.136619 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.139659 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.139639 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 15:28:11.139777 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.139699 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q7qvb\"" Jul 06 15:28:11.140061 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.140049 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 15:28:11.155218 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.155194 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9s9qg"] Jul 06 15:28:11.199637 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.199616 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 15:28:11.200028 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.200015 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/0.log" Jul 06 15:28:11.200069 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.200047 2561 generic.go:356] "Generic (PLEG): container finished" podID="6283d26a-0676-4b9e-be5b-7a41a44079ae" containerID="ce5becad4940311fde083454a9fea43bb8dcbcd1dda433b018826b640c8fd2ee" exitCode=255 Jul 06 15:28:11.200136 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.200119 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" event={"ID":"6283d26a-0676-4b9e-be5b-7a41a44079ae","Type":"ContainerDied","Data":"ce5becad4940311fde083454a9fea43bb8dcbcd1dda433b018826b640c8fd2ee"} Jul 06 15:28:11.200180 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.200169 2561 scope.go:117] "RemoveContainer" containerID="287d673977c437b6c927989050fd523962356e4ef2f89d8c7233bf392b6dc8cc" Jul 06 15:28:11.200359 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.200344 2561 scope.go:117] "RemoveContainer" containerID="ce5becad4940311fde083454a9fea43bb8dcbcd1dda433b018826b640c8fd2ee" Jul 06 15:28:11.200573 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:11.200540 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-q2lgf_openshift-console-operator(6283d26a-0676-4b9e-be5b-7a41a44079ae)\"" pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" podUID="6283d26a-0676-4b9e-be5b-7a41a44079ae" Jul 06 15:28:11.294669 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.294599 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-data-volume\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.294780 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.294666 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-crio-socket\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.294975 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.294957 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.295040 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.295001 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4rx6\" (UniqueName: \"kubernetes.io/projected/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-api-access-c4rx6\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.295040 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.295020 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.320683 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.320650 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:11.395987 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.395960 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-data-volume\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.396133 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.395998 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-crio-socket\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.396133 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.396073 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-crio-socket\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.396133 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.396105 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.396271 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.396148 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4rx6\" (UniqueName: \"kubernetes.io/projected/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-api-access-c4rx6\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.396271 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.396171 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.396271 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:11.396246 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:28:11.396359 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.396281 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-data-volume\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.396359 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:11.396306 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 15:28:11.396359 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:11.396308 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls podName:c63b1ff2-df2d-4354-9707-4bee7aa4ccb3 nodeName:}" failed. No retries permitted until 2026-07-06 15:28:11.896294085 +0000 UTC m=+113.528846049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9s9qg" (UID: "c63b1ff2-df2d-4354-9707-4bee7aa4ccb3") : secret "insights-runtime-extractor-tls" not found Jul 06 15:28:11.396456 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:11.396373 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm podName:c63b1ff2-df2d-4354-9707-4bee7aa4ccb3 nodeName:}" failed. No retries permitted until 2026-07-06 15:28:11.896356361 +0000 UTC m=+113.528908324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9s9qg" (UID: "c63b1ff2-df2d-4354-9707-4bee7aa4ccb3") : configmap "kube-rbac-proxy" not found Jul 06 15:28:11.405182 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.405163 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4rx6\" (UniqueName: \"kubernetes.io/projected/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-api-access-c4rx6\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.900423 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.900394 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.900577 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:11.900444 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:11.900577 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:11.900523 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:28:11.900671 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:11.900593 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls podName:c63b1ff2-df2d-4354-9707-4bee7aa4ccb3 nodeName:}" failed. No retries permitted until 2026-07-06 15:28:12.90057924 +0000 UTC m=+114.533131204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9s9qg" (UID: "c63b1ff2-df2d-4354-9707-4bee7aa4ccb3") : secret "insights-runtime-extractor-tls" not found Jul 06 15:28:11.900671 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:11.900627 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 15:28:11.900752 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:11.900681 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm podName:c63b1ff2-df2d-4354-9707-4bee7aa4ccb3 nodeName:}" failed. No retries permitted until 2026-07-06 15:28:12.900668903 +0000 UTC m=+114.533220865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9s9qg" (UID: "c63b1ff2-df2d-4354-9707-4bee7aa4ccb3") : configmap "kube-rbac-proxy" not found Jul 06 15:28:12.205180 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.205112 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 15:28:12.205550 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.205424 2561 scope.go:117] "RemoveContainer" containerID="ce5becad4940311fde083454a9fea43bb8dcbcd1dda433b018826b640c8fd2ee" Jul 06 15:28:12.205609 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:12.205590 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-q2lgf_openshift-console-operator(6283d26a-0676-4b9e-be5b-7a41a44079ae)\"" pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" podUID="6283d26a-0676-4b9e-be5b-7a41a44079ae" Jul 06 15:28:12.761952 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.761915 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-6548r"] Jul 06 15:28:12.766008 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.765993 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-6548r" Jul 06 15:28:12.769574 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.769551 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 06 15:28:12.769574 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.769568 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cdr49\"" Jul 06 15:28:12.770679 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.770661 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 06 15:28:12.770952 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.770928 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 06 15:28:12.770952 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.770937 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 06 15:28:12.775403 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.775383 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-6548r"] Jul 06 15:28:12.809073 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.809050 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ls2rn\" (UID: \"17fdbe64-b3a3-42b9-91eb-68b74152fdda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" Jul 06 15:28:12.809193 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:12.809180 2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 15:28:12.809247 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:12.809238 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls podName:17fdbe64-b3a3-42b9-91eb-68b74152fdda nodeName:}" failed. No retries permitted until 2026-07-06 15:28:20.809225709 +0000 UTC m=+122.441777670 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-ls2rn" (UID: "17fdbe64-b3a3-42b9-91eb-68b74152fdda") : secret "samples-operator-tls" not found Jul 06 15:28:12.866336 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.866301 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:12.909382 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.909361 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:12.909476 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.909402 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f9tz\" (UniqueName: \"kubernetes.io/projected/e244ad33-c670-4ed5-a9b6-713a4bff1edb-kube-api-access-4f9tz\") pod \"service-ca-758f599c5-6548r\" (UID: \"e244ad33-c670-4ed5-a9b6-713a4bff1edb\") " pod="openshift-service-ca/service-ca-758f599c5-6548r" Jul 06 15:28:12.909548 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:12.909481 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 15:28:12.909598 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:12.909555 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm podName:c63b1ff2-df2d-4354-9707-4bee7aa4ccb3 nodeName:}" failed. No retries permitted until 2026-07-06 15:28:14.909519527 +0000 UTC m=+116.542071489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9s9qg" (UID: "c63b1ff2-df2d-4354-9707-4bee7aa4ccb3") : configmap "kube-rbac-proxy" not found Jul 06 15:28:12.909598 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.909577 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e244ad33-c670-4ed5-a9b6-713a4bff1edb-signing-key\") pod \"service-ca-758f599c5-6548r\" (UID: \"e244ad33-c670-4ed5-a9b6-713a4bff1edb\") " pod="openshift-service-ca/service-ca-758f599c5-6548r" Jul 06 15:28:12.909704 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.909603 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e244ad33-c670-4ed5-a9b6-713a4bff1edb-signing-cabundle\") pod \"service-ca-758f599c5-6548r\" (UID: \"e244ad33-c670-4ed5-a9b6-713a4bff1edb\") " pod="openshift-service-ca/service-ca-758f599c5-6548r" Jul 06 15:28:12.909704 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.909678 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:12.909854 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:12.909764 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:28:12.909854 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:12.909815 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls podName:c63b1ff2-df2d-4354-9707-4bee7aa4ccb3 nodeName:}" failed. No retries permitted until 2026-07-06 15:28:14.909798776 +0000 UTC m=+116.542350736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9s9qg" (UID: "c63b1ff2-df2d-4354-9707-4bee7aa4ccb3") : secret "insights-runtime-extractor-tls" not found Jul 06 15:28:12.971216 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:12.971178 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:13.010154 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:13.010134 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e244ad33-c670-4ed5-a9b6-713a4bff1edb-signing-cabundle\") pod \"service-ca-758f599c5-6548r\" (UID: \"e244ad33-c670-4ed5-a9b6-713a4bff1edb\") " pod="openshift-service-ca/service-ca-758f599c5-6548r" Jul 06 15:28:13.010248 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:13.010213 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f9tz\" (UniqueName: \"kubernetes.io/projected/e244ad33-c670-4ed5-a9b6-713a4bff1edb-kube-api-access-4f9tz\") pod \"service-ca-758f599c5-6548r\" (UID: \"e244ad33-c670-4ed5-a9b6-713a4bff1edb\") " pod="openshift-service-ca/service-ca-758f599c5-6548r" Jul 06 15:28:13.010288 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:13.010269 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e244ad33-c670-4ed5-a9b6-713a4bff1edb-signing-key\") pod \"service-ca-758f599c5-6548r\" (UID: \"e244ad33-c670-4ed5-a9b6-713a4bff1edb\") " pod="openshift-service-ca/service-ca-758f599c5-6548r" Jul 06 15:28:13.010866 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:13.010846 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e244ad33-c670-4ed5-a9b6-713a4bff1edb-signing-cabundle\") pod \"service-ca-758f599c5-6548r\" (UID: \"e244ad33-c670-4ed5-a9b6-713a4bff1edb\") " pod="openshift-service-ca/service-ca-758f599c5-6548r" Jul 06 15:28:13.012711 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:13.012658 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e244ad33-c670-4ed5-a9b6-713a4bff1edb-signing-key\") pod \"service-ca-758f599c5-6548r\" (UID: \"e244ad33-c670-4ed5-a9b6-713a4bff1edb\") " pod="openshift-service-ca/service-ca-758f599c5-6548r" Jul 06 15:28:13.019874 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:13.019853 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f9tz\" (UniqueName: \"kubernetes.io/projected/e244ad33-c670-4ed5-a9b6-713a4bff1edb-kube-api-access-4f9tz\") pod \"service-ca-758f599c5-6548r\" (UID: \"e244ad33-c670-4ed5-a9b6-713a4bff1edb\") " pod="openshift-service-ca/service-ca-758f599c5-6548r" Jul 06 15:28:13.074752 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:13.074729 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-6548r" Jul 06 15:28:13.198814 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:13.198791 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-6548r"] Jul 06 15:28:13.201029 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:13.201003 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode244ad33_c670_4ed5_a9b6_713a4bff1edb.slice/crio-605dc05fe636e101220c2c50154ef8894bf8770cd869f1235d4875fc462bc4eb WatchSource:0}: Error finding container 605dc05fe636e101220c2c50154ef8894bf8770cd869f1235d4875fc462bc4eb: Status 404 returned error can't find the container with id 605dc05fe636e101220c2c50154ef8894bf8770cd869f1235d4875fc462bc4eb Jul 06 15:28:13.209163 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:13.209142 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-6548r" event={"ID":"e244ad33-c670-4ed5-a9b6-713a4bff1edb","Type":"ContainerStarted","Data":"605dc05fe636e101220c2c50154ef8894bf8770cd869f1235d4875fc462bc4eb"} Jul 06 15:28:13.847470 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:13.847423 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:14.121236 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:14.121160 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w8l5z_75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6/dns-node-resolver/0.log" Jul 06 15:28:14.213983 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:14.213952 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-6548r" event={"ID":"e244ad33-c670-4ed5-a9b6-713a4bff1edb","Type":"ContainerStarted","Data":"c27f2576b9ce6f6ebbf10b91c4891da24a79f2c955951e43c347f8b82318f9a5"} Jul 06 15:28:14.923195 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:14.923158 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:14.923374 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:14.923201 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:14.923374 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:14.923294 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 15:28:14.923374 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:14.923298 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:28:14.923374 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:14.923344 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm podName:c63b1ff2-df2d-4354-9707-4bee7aa4ccb3 nodeName:}" failed. No retries permitted until 2026-07-06 15:28:18.923330882 +0000 UTC m=+120.555882843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9s9qg" (UID: "c63b1ff2-df2d-4354-9707-4bee7aa4ccb3") : configmap "kube-rbac-proxy" not found Jul 06 15:28:14.923374 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:14.923356 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls podName:c63b1ff2-df2d-4354-9707-4bee7aa4ccb3 nodeName:}" failed. No retries permitted until 2026-07-06 15:28:18.923350613 +0000 UTC m=+120.555902573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9s9qg" (UID: "c63b1ff2-df2d-4354-9707-4bee7aa4ccb3") : secret "insights-runtime-extractor-tls" not found Jul 06 15:28:15.121947 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:15.121919 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w44v2_3b2617bf-d392-4486-ab8c-3d0aa5f481aa/node-ca/0.log" Jul 06 15:28:15.410323 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:15.410290 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:15.457231 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:15.457191 2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:15.457414 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:15.457261 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:15.457809 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:15.457792 2561 scope.go:117] "RemoveContainer" containerID="ce5becad4940311fde083454a9fea43bb8dcbcd1dda433b018826b640c8fd2ee" Jul 06 15:28:15.458042 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:15.458017 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-q2lgf_openshift-console-operator(6283d26a-0676-4b9e-be5b-7a41a44079ae)\"" pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" podUID="6283d26a-0676-4b9e-be5b-7a41a44079ae" Jul 06 15:28:15.628026 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:15.627993 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:15.773906 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:15.773856 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:15.892297 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:15.892266 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:15.920231 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:15.920204 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lrpnb_fffc4cf2-1b41-40c5-943f-f01cc52933b7/serve-healthcheck-canary/0.log" Jul 06 15:28:16.108605 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:16.108500 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:16.522059 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:16.522027 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-px9pj_f6dbf566-cef0-4ead-a469-1a197bce5b3d/kube-storage-version-migrator-operator/0.log" Jul 06 15:28:17.310427 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:17.310392 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:17.507019 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:17.506979 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:17.679843 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:17.679763 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:17.882230 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:17.882178 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:18.088680 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:18.088644 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:18.885288 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:18.885250 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:18.958315 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:18.957966 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:18.958315 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:18.958089 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 15:28:18.958315 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:18.958113 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:18.958315 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:18.958162 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm podName:c63b1ff2-df2d-4354-9707-4bee7aa4ccb3 nodeName:}" failed. No retries permitted until 2026-07-06 15:28:26.958140584 +0000 UTC m=+128.590692547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9s9qg" (UID: "c63b1ff2-df2d-4354-9707-4bee7aa4ccb3") : configmap "kube-rbac-proxy" not found Jul 06 15:28:18.958315 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:18.958236 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:28:18.958315 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:18.958289 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls podName:c63b1ff2-df2d-4354-9707-4bee7aa4ccb3 nodeName:}" failed. No retries permitted until 2026-07-06 15:28:26.958272577 +0000 UTC m=+128.590824545 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9s9qg" (UID: "c63b1ff2-df2d-4354-9707-4bee7aa4ccb3") : secret "insights-runtime-extractor-tls" not found Jul 06 15:28:19.193002 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:19.192910 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:19.351884 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:19.351850 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:19.504634 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:19.504604 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:19.696458 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:19.696424 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:20.113901 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:20.113871 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:20.219436 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:20.219410 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:20.306198 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:20.306164 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:20.872188 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:20.872156 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ls2rn\" (UID: \"17fdbe64-b3a3-42b9-91eb-68b74152fdda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" Jul 06 15:28:20.874707 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:20.874688 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/17fdbe64-b3a3-42b9-91eb-68b74152fdda-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-ls2rn\" (UID: \"17fdbe64-b3a3-42b9-91eb-68b74152fdda\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" Jul 06 15:28:20.894897 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:20.894870 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:20.940576 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:20.940552 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gjcvt\"" Jul 06 15:28:20.948448 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:20.948430 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" Jul 06 15:28:21.085685 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:21.085637 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-6548r" podStartSLOduration=9.085620521 podStartE2EDuration="9.085620521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:14.238055884 +0000 UTC m=+115.870607866" watchObservedRunningTime="2026-07-06 15:28:21.085620521 +0000 UTC m=+122.718172504" Jul 06 15:28:21.085989 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:21.085972 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn"] Jul 06 15:28:21.237655 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:21.237619 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" event={"ID":"17fdbe64-b3a3-42b9-91eb-68b74152fdda","Type":"ContainerStarted","Data":"89354d85aab6156a9df6c0ac73a430868415bc7d06875c41d10a2c0166a03eb6"} Jul 06 15:28:21.503763 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:21.503732 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:21.712112 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:21.712061 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:21.869605 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:21.868993 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:21.969892 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:21.969858 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:22.291990 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:22.291953 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:22.504061 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:22.504029 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:23.247282 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:23.247249 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" event={"ID":"17fdbe64-b3a3-42b9-91eb-68b74152fdda","Type":"ContainerStarted","Data":"7db30ec668d20d432a050c90178c8ab4eee71e0ca9cbabda62dfdaf91c9a2eaa"} Jul 06 15:28:23.247448 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:23.247296 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" event={"ID":"17fdbe64-b3a3-42b9-91eb-68b74152fdda","Type":"ContainerStarted","Data":"21ecbe8d4d0ecf1c38b01799b46a0f599e3af8ef290866ed86e2b4c81691dda3"} Jul 06 15:28:23.271017 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:23.270966 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-ls2rn" podStartSLOduration=16.792057859 podStartE2EDuration="18.270953202s" totalImagesPullingTime="1.478895343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:23.269482019 +0000 UTC m=+124.902034002" watchObservedRunningTime="2026-07-06 15:28:23.270953202 +0000 UTC m=+124.903505184" Jul 06 15:28:23.686817 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:23.686727 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:23.879584 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:23.879553 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:23.979445 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:23.979415 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:24.120598 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:24.120573 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:24.315852 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:24.315785 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:24.487558 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:24.487516 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:25.490654 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:25.490617 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:25.664771 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:25.664741 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:25.737380 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:25.737351 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:25.872208 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:25.872138 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:26.074224 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:26.074195 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:27.019623 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:27.019590 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:27.020004 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:27.019630 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:27.020112 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:27.020095 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:27.022025 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:27.022001 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c63b1ff2-df2d-4354-9707-4bee7aa4ccb3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qg\" (UID: \"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3\") " pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:27.049232 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:27.049206 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q7qvb\"" Jul 06 15:28:27.057123 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:27.057094 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9s9qg" Jul 06 15:28:27.194377 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:27.194300 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9s9qg"] Jul 06 15:28:27.196482 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:27.196461 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63b1ff2_df2d_4354_9707_4bee7aa4ccb3.slice/crio-95c4decdca666bb672b9f3d8d162e16be69d2f6ce56a9429c4dfe916183ff1f2 WatchSource:0}: Error finding container 95c4decdca666bb672b9f3d8d162e16be69d2f6ce56a9429c4dfe916183ff1f2: Status 404 returned error can't find the container with id 95c4decdca666bb672b9f3d8d162e16be69d2f6ce56a9429c4dfe916183ff1f2 Jul 06 15:28:27.260161 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:27.260136 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9s9qg" event={"ID":"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3","Type":"ContainerStarted","Data":"95c4decdca666bb672b9f3d8d162e16be69d2f6ce56a9429c4dfe916183ff1f2"} Jul 06 15:28:28.633470 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:28.633418 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs\") pod \"network-metrics-daemon-m4llb\" (UID: \"d047efe8-bfb0-4fee-a069-15a344dc4726\") " pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:28:28.636022 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:28.635998 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d047efe8-bfb0-4fee-a069-15a344dc4726-metrics-certs\") pod \"network-metrics-daemon-m4llb\" (UID: \"d047efe8-bfb0-4fee-a069-15a344dc4726\") " pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:28:28.828082 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:28.828053 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-km2tx\"" Jul 06 15:28:28.835178 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:28.835157 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m4llb" Jul 06 15:28:28.987776 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:28.987731 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:29.114148 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:29.114121 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:29.793726 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:29.793695 2561 scope.go:117] "RemoveContainer" containerID="ce5becad4940311fde083454a9fea43bb8dcbcd1dda433b018826b640c8fd2ee" Jul 06 15:28:31.055553 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:31.055466 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m4llb"] Jul 06 15:28:31.060089 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:31.060061 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd047efe8_bfb0_4fee_a069_15a344dc4726.slice/crio-5234817e461fb2975cda63457ea94bcf58e232b26192078fce161408a23fa008 WatchSource:0}: Error finding container 5234817e461fb2975cda63457ea94bcf58e232b26192078fce161408a23fa008: Status 404 returned error can't find the container with id 5234817e461fb2975cda63457ea94bcf58e232b26192078fce161408a23fa008 Jul 06 15:28:31.276125 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:31.276088 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9s9qg" event={"ID":"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3","Type":"ContainerStarted","Data":"ac1a1d634b68cc7d25b0d2636b55ea7e49f697804f650def4bed6eab535bfc7b"} Jul 06 15:28:31.277322 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:31.277295 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m4llb" event={"ID":"d047efe8-bfb0-4fee-a069-15a344dc4726","Type":"ContainerStarted","Data":"5234817e461fb2975cda63457ea94bcf58e232b26192078fce161408a23fa008"} Jul 06 15:28:31.279031 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:31.279014 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 15:28:31.279127 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:31.279060 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" event={"ID":"6283d26a-0676-4b9e-be5b-7a41a44079ae","Type":"ContainerStarted","Data":"27b523511bb9750edecf7ad55695ccf7fc1fd778bc01585a8c9d6d4aa495d0fb"} Jul 06 15:28:31.279379 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:31.279359 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:31.308643 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:31.308599 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" podStartSLOduration=22.708062079 podStartE2EDuration="26.308586433s" totalImagesPullingTime="3.600524354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:31.308229377 +0000 UTC m=+132.940781362" watchObservedRunningTime="2026-07-06 15:28:31.308586433 +0000 UTC m=+132.941138454" Jul 06 15:28:31.357357 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:31.357332 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-q2lgf" Jul 06 15:28:31.602300 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:31.602227 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:31.724014 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:31.723985 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:32.102915 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:32.102877 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:32.284828 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:32.284798 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9s9qg" event={"ID":"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3","Type":"ContainerStarted","Data":"ba5e931bbb2059e0b770dca24e5494ed1e4272609c5219d72002a98676e3c010"} Jul 06 15:28:32.286166 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:32.286102 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m4llb" event={"ID":"d047efe8-bfb0-4fee-a069-15a344dc4726","Type":"ContainerStarted","Data":"9dd43211c787c5975f2efef738dfe835e3cf4db2c937317ef61409fa56010396"} Jul 06 15:28:32.319987 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:32.319941 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:32.460690 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:32.460658 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:32.655992 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:32.655906 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:32.779308 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:32.779274 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:32.892653 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:32.892615 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:33.292371 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:33.292333 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m4llb" event={"ID":"d047efe8-bfb0-4fee-a069-15a344dc4726","Type":"ContainerStarted","Data":"a5561006f90c0a19314a8133fe182a4db96e21b3d9760eee0c2e19decb090d31"} Jul 06 15:28:33.315861 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:33.315808 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m4llb" podStartSLOduration=133.182957891 podStartE2EDuration="2m14.315790368s" totalImagesPullingTime="1.132832477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:33.314768393 +0000 UTC m=+134.947320377" watchObservedRunningTime="2026-07-06 15:28:33.315790368 +0000 UTC m=+134.948342353" Jul 06 15:28:33.501717 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:33.501682 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:33.655728 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:33.655609 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:33.840593 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:33.840556 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:34.477043 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:34.477017 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:34.576342 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:34.576312 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:35.301615 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:35.301580 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9s9qg" event={"ID":"c63b1ff2-df2d-4354-9707-4bee7aa4ccb3","Type":"ContainerStarted","Data":"31e4831b2ced0b2f94986eabe3a42314d92b0545b803e6fd822b09ee35cd5293"} Jul 06 15:28:35.323021 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:35.322979 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9s9qg" podStartSLOduration=17.19294458 podStartE2EDuration="24.322964841s" totalImagesPullingTime="7.130020261s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:28:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:35.322051517 +0000 UTC m=+136.954603501" watchObservedRunningTime="2026-07-06 15:28:35.322964841 +0000 UTC m=+136.955516825" Jul 06 15:28:36.407766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.407732 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-2qq7v"] Jul 06 15:28:36.411849 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.411828 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-2qq7v" Jul 06 15:28:36.414802 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.414773 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 15:28:36.414915 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.414865 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 15:28:36.414974 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.414953 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-4k2pq\"" Jul 06 15:28:36.429171 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.429140 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-2qq7v"] Jul 06 15:28:36.493297 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.493265 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9p2v\" (UniqueName: \"kubernetes.io/projected/55501abc-cc50-4838-b8ac-c28284b253f0-kube-api-access-v9p2v\") pod \"downloads-685bfb5f64-2qq7v\" (UID: \"55501abc-cc50-4838-b8ac-c28284b253f0\") " pod="openshift-console/downloads-685bfb5f64-2qq7v" Jul 06 15:28:36.494750 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.494704 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:36.507659 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.507639 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c6fd88d46-wfqsl"] Jul 06 15:28:36.510599 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.510584 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.513830 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.513816 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 15:28:36.513942 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.513922 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 15:28:36.514038 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.514024 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pngzd\"" Jul 06 15:28:36.514269 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.514255 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 15:28:36.519410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.519394 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 15:28:36.536058 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.536036 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c6fd88d46-wfqsl"] Jul 06 15:28:36.594715 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.594686 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9p2v\" (UniqueName: \"kubernetes.io/projected/55501abc-cc50-4838-b8ac-c28284b253f0-kube-api-access-v9p2v\") pod \"downloads-685bfb5f64-2qq7v\" (UID: \"55501abc-cc50-4838-b8ac-c28284b253f0\") " pod="openshift-console/downloads-685bfb5f64-2qq7v" Jul 06 15:28:36.594859 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.594737 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13d38e19-fbaf-4051-9062-64ab5d08eea6-trusted-ca\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.594859 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.594769 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13d38e19-fbaf-4051-9062-64ab5d08eea6-image-registry-private-configuration\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.594859 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.594830 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13d38e19-fbaf-4051-9062-64ab5d08eea6-ca-trust-extracted\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.595003 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.594901 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfb7b\" (UniqueName: \"kubernetes.io/projected/13d38e19-fbaf-4051-9062-64ab5d08eea6-kube-api-access-qfb7b\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.595003 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.594937 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13d38e19-fbaf-4051-9062-64ab5d08eea6-registry-tls\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.595003 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.594958 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13d38e19-fbaf-4051-9062-64ab5d08eea6-registry-certificates\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.595003 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.594995 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13d38e19-fbaf-4051-9062-64ab5d08eea6-bound-sa-token\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.595180 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.595053 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13d38e19-fbaf-4051-9062-64ab5d08eea6-installation-pull-secrets\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.603986 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.603959 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9p2v\" (UniqueName: \"kubernetes.io/projected/55501abc-cc50-4838-b8ac-c28284b253f0-kube-api-access-v9p2v\") pod \"downloads-685bfb5f64-2qq7v\" (UID: \"55501abc-cc50-4838-b8ac-c28284b253f0\") " pod="openshift-console/downloads-685bfb5f64-2qq7v" Jul 06 15:28:36.651663 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.651638 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:36.695720 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.695665 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13d38e19-fbaf-4051-9062-64ab5d08eea6-trusted-ca\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.695720 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.695693 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13d38e19-fbaf-4051-9062-64ab5d08eea6-image-registry-private-configuration\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.695866 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.695734 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13d38e19-fbaf-4051-9062-64ab5d08eea6-ca-trust-extracted\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.695866 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.695847 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfb7b\" (UniqueName: \"kubernetes.io/projected/13d38e19-fbaf-4051-9062-64ab5d08eea6-kube-api-access-qfb7b\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.695964 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.695879 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13d38e19-fbaf-4051-9062-64ab5d08eea6-registry-tls\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.695964 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.695935 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13d38e19-fbaf-4051-9062-64ab5d08eea6-registry-certificates\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.696062 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.695972 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13d38e19-fbaf-4051-9062-64ab5d08eea6-bound-sa-token\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.696062 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.695999 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13d38e19-fbaf-4051-9062-64ab5d08eea6-installation-pull-secrets\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.696160 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.696095 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13d38e19-fbaf-4051-9062-64ab5d08eea6-ca-trust-extracted\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.696833 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.696810 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13d38e19-fbaf-4051-9062-64ab5d08eea6-trusted-ca\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.697117 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.697092 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13d38e19-fbaf-4051-9062-64ab5d08eea6-registry-certificates\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.698240 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.698206 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13d38e19-fbaf-4051-9062-64ab5d08eea6-image-registry-private-configuration\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.698450 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.698432 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13d38e19-fbaf-4051-9062-64ab5d08eea6-registry-tls\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.698557 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.698541 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13d38e19-fbaf-4051-9062-64ab5d08eea6-installation-pull-secrets\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.710092 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.710069 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfb7b\" (UniqueName: \"kubernetes.io/projected/13d38e19-fbaf-4051-9062-64ab5d08eea6-kube-api-access-qfb7b\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.711138 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.711121 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13d38e19-fbaf-4051-9062-64ab5d08eea6-bound-sa-token\") pod \"image-registry-7c6fd88d46-wfqsl\" (UID: \"13d38e19-fbaf-4051-9062-64ab5d08eea6\") " pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.722034 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.722015 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-2qq7v" Jul 06 15:28:36.821678 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.821652 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:36.865661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.865622 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-2qq7v"] Jul 06 15:28:36.868670 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:36.868647 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55501abc_cc50_4838_b8ac_c28284b253f0.slice/crio-75fe3a21098130dc39cab21e756dcbe33e46feee38e5f0a88a822e3711e870d7 WatchSource:0}: Error finding container 75fe3a21098130dc39cab21e756dcbe33e46feee38e5f0a88a822e3711e870d7: Status 404 returned error can't find the container with id 75fe3a21098130dc39cab21e756dcbe33e46feee38e5f0a88a822e3711e870d7 Jul 06 15:28:36.957889 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:36.957865 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c6fd88d46-wfqsl"] Jul 06 15:28:36.960033 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:36.959997 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d38e19_fbaf_4051_9062_64ab5d08eea6.slice/crio-f48f63ecdf749e63f5f979d6837b4fa0adbfbd3fbcbec63cee30a45245a1ce53 WatchSource:0}: Error finding container f48f63ecdf749e63f5f979d6837b4fa0adbfbd3fbcbec63cee30a45245a1ce53: Status 404 returned error can't find the container with id f48f63ecdf749e63f5f979d6837b4fa0adbfbd3fbcbec63cee30a45245a1ce53 Jul 06 15:28:37.310096 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:37.310062 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" event={"ID":"13d38e19-fbaf-4051-9062-64ab5d08eea6","Type":"ContainerStarted","Data":"c37d1076c3c70ec13af46c86dff536500b846c7565a71a27a5628b6323afee32"} Jul 06 15:28:37.310292 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:37.310111 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" event={"ID":"13d38e19-fbaf-4051-9062-64ab5d08eea6","Type":"ContainerStarted","Data":"f48f63ecdf749e63f5f979d6837b4fa0adbfbd3fbcbec63cee30a45245a1ce53"} Jul 06 15:28:37.310292 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:37.310174 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:37.311279 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:37.311254 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-2qq7v" event={"ID":"55501abc-cc50-4838-b8ac-c28284b253f0","Type":"ContainerStarted","Data":"75fe3a21098130dc39cab21e756dcbe33e46feee38e5f0a88a822e3711e870d7"} Jul 06 15:28:37.335945 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:37.335894 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" podStartSLOduration=1.335881921 podStartE2EDuration="1.335881921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:37.333978534 +0000 UTC m=+138.966530518" watchObservedRunningTime="2026-07-06 15:28:37.335881921 +0000 UTC m=+138.968433904" Jul 06 15:28:38.698427 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:38.698390 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:38.797868 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:38.797836 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:39.164965 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:39.164928 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:39.263726 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:39.263694 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:39.353002 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:39.352968 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:39.626268 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:39.626236 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:42.406320 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.406280 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:42.436163 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.436137 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw"] Jul 06 15:28:42.466314 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.466289 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw"] Jul 06 15:28:42.466455 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.466410 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" Jul 06 15:28:42.472350 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.472274 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 15:28:42.472494 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.472475 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 15:28:42.477661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.473882 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 15:28:42.477661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.474603 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 06 15:28:42.477661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.475268 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 06 15:28:42.477661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.475719 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-cxdwv\"" Jul 06 15:28:42.496665 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.496635 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:42.543010 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.542982 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb408066-218e-4efb-8407-ccf6c8df3994-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-r6mpw\" (UID: \"cb408066-218e-4efb-8407-ccf6c8df3994\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" Jul 06 15:28:42.543123 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.543011 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb408066-218e-4efb-8407-ccf6c8df3994-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-r6mpw\" (UID: \"cb408066-218e-4efb-8407-ccf6c8df3994\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" Jul 06 15:28:42.543123 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.543036 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb408066-218e-4efb-8407-ccf6c8df3994-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-r6mpw\" (UID: \"cb408066-218e-4efb-8407-ccf6c8df3994\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" Jul 06 15:28:42.543123 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.543060 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6p64\" (UniqueName: \"kubernetes.io/projected/cb408066-218e-4efb-8407-ccf6c8df3994-kube-api-access-r6p64\") pod \"prometheus-operator-567f7b8b7c-r6mpw\" (UID: \"cb408066-218e-4efb-8407-ccf6c8df3994\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" Jul 06 15:28:42.643659 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.643616 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb408066-218e-4efb-8407-ccf6c8df3994-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-r6mpw\" (UID: \"cb408066-218e-4efb-8407-ccf6c8df3994\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" Jul 06 15:28:42.643832 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.643670 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb408066-218e-4efb-8407-ccf6c8df3994-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-r6mpw\" (UID: \"cb408066-218e-4efb-8407-ccf6c8df3994\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" Jul 06 15:28:42.643832 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.643714 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb408066-218e-4efb-8407-ccf6c8df3994-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-r6mpw\" (UID: \"cb408066-218e-4efb-8407-ccf6c8df3994\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" Jul 06 15:28:42.643832 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.643756 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6p64\" (UniqueName: \"kubernetes.io/projected/cb408066-218e-4efb-8407-ccf6c8df3994-kube-api-access-r6p64\") pod \"prometheus-operator-567f7b8b7c-r6mpw\" (UID: \"cb408066-218e-4efb-8407-ccf6c8df3994\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" Jul 06 15:28:42.643832 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:42.643814 2561 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 06 15:28:42.644058 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:42.643902 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb408066-218e-4efb-8407-ccf6c8df3994-prometheus-operator-tls podName:cb408066-218e-4efb-8407-ccf6c8df3994 nodeName:}" failed. No retries permitted until 2026-07-06 15:28:43.143880312 +0000 UTC m=+144.776432284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/cb408066-218e-4efb-8407-ccf6c8df3994-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-r6mpw" (UID: "cb408066-218e-4efb-8407-ccf6c8df3994") : secret "prometheus-operator-tls" not found Jul 06 15:28:42.644517 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.644493 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb408066-218e-4efb-8407-ccf6c8df3994-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-r6mpw\" (UID: \"cb408066-218e-4efb-8407-ccf6c8df3994\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" Jul 06 15:28:42.646931 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.646907 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb408066-218e-4efb-8407-ccf6c8df3994-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-r6mpw\" (UID: \"cb408066-218e-4efb-8407-ccf6c8df3994\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" Jul 06 15:28:42.653952 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.653930 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6p64\" (UniqueName: \"kubernetes.io/projected/cb408066-218e-4efb-8407-ccf6c8df3994-kube-api-access-r6p64\") pod \"prometheus-operator-567f7b8b7c-r6mpw\" (UID: \"cb408066-218e-4efb-8407-ccf6c8df3994\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" Jul 06 15:28:42.870309 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:42.870273 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:43.148262 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:43.148167 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb408066-218e-4efb-8407-ccf6c8df3994-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-r6mpw\" (UID: \"cb408066-218e-4efb-8407-ccf6c8df3994\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" Jul 06 15:28:43.151348 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:43.151321 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb408066-218e-4efb-8407-ccf6c8df3994-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-r6mpw\" (UID: \"cb408066-218e-4efb-8407-ccf6c8df3994\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" Jul 06 15:28:43.379357 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:43.379325 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" Jul 06 15:28:43.538278 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:43.538244 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw"] Jul 06 15:28:43.540824 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:43.540789 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb408066_218e_4efb_8407_ccf6c8df3994.slice/crio-682c8589a4c8484b245e8129810c55ffc221e6bf7f4bf161b1c0cd1c2287ff7f WatchSource:0}: Error finding container 682c8589a4c8484b245e8129810c55ffc221e6bf7f4bf161b1c0cd1c2287ff7f: Status 404 returned error can't find the container with id 682c8589a4c8484b245e8129810c55ffc221e6bf7f4bf161b1c0cd1c2287ff7f Jul 06 15:28:44.337910 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:44.337867 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" event={"ID":"cb408066-218e-4efb-8407-ccf6c8df3994","Type":"ContainerStarted","Data":"682c8589a4c8484b245e8129810c55ffc221e6bf7f4bf161b1c0cd1c2287ff7f"} Jul 06 15:28:45.274987 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.273978 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9d7bff7f-n5zfn"] Jul 06 15:28:45.276709 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.276682 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.281081 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.281022 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 15:28:45.281208 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.281168 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 15:28:45.282933 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.282679 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 15:28:45.282933 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.282916 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 15:28:45.283137 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.282966 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 15:28:45.283137 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.283009 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rmsbt\"" Jul 06 15:28:45.290457 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.290396 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d7bff7f-n5zfn"] Jul 06 15:28:45.344135 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.344089 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" event={"ID":"cb408066-218e-4efb-8407-ccf6c8df3994","Type":"ContainerStarted","Data":"3bced8d55d21dcc14d32205daab9cc8d6193c8641a27ad34774898be50dfb563"} Jul 06 15:28:45.344291 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.344150 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" event={"ID":"cb408066-218e-4efb-8407-ccf6c8df3994","Type":"ContainerStarted","Data":"b3d7cd3fe78894566d70ac201ac8e352117ac2637b708c218f26ab5556f685fe"} Jul 06 15:28:45.360963 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.360938 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:45.367583 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.367517 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-r6mpw" podStartSLOduration=1.8800459630000002 podStartE2EDuration="3.367498001s" totalImagesPullingTime="1.487452038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:45.366411823 +0000 UTC m=+146.998963808" watchObservedRunningTime="2026-07-06 15:28:45.367498001 +0000 UTC m=+147.000049985" Jul 06 15:28:45.367729 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.367672 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-oauth-config\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.367794 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.367743 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-serving-cert\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.367794 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.367767 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn8s9\" (UniqueName: \"kubernetes.io/projected/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-kube-api-access-kn8s9\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.367895 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.367853 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-service-ca\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.367948 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.367909 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-oauth-serving-cert\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.367948 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.367934 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-config\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.468887 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.468858 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-oauth-config\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.469034 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.468899 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-serving-cert\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.469034 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.468926 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn8s9\" (UniqueName: \"kubernetes.io/projected/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-kube-api-access-kn8s9\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.469034 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.468957 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-service-ca\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.469034 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.468988 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-oauth-serving-cert\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.469034 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.469012 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-config\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.469820 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.469794 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-config\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.469939 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.469824 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-service-ca\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.469939 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.469845 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-oauth-serving-cert\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.471834 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.471812 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-serving-cert\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.471931 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.471865 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-oauth-config\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.479053 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.479034 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn8s9\" (UniqueName: \"kubernetes.io/projected/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-kube-api-access-kn8s9\") pod \"console-9d7bff7f-n5zfn\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.594027 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.593908 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:28:45.745891 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:45.745856 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d7bff7f-n5zfn"] Jul 06 15:28:45.748431 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:45.748401 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94cdbae_fb8c_404a_bdf4_6fa955bccab9.slice/crio-e6f4c009900f979126d24f6e174105e1562e267c03f6a0f888a820dbde9be737 WatchSource:0}: Error finding container e6f4c009900f979126d24f6e174105e1562e267c03f6a0f888a820dbde9be737: Status 404 returned error can't find the container with id e6f4c009900f979126d24f6e174105e1562e267c03f6a0f888a820dbde9be737 Jul 06 15:28:46.044803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.044771 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:46.126397 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.126365 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:46.350116 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.350018 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d7bff7f-n5zfn" event={"ID":"d94cdbae-fb8c-404a-bdf4-6fa955bccab9","Type":"ContainerStarted","Data":"e6f4c009900f979126d24f6e174105e1562e267c03f6a0f888a820dbde9be737"} Jul 06 15:28:46.549011 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.548978 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:46.787668 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.787590 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:46.819725 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.819695 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5wrkh"] Jul 06 15:28:46.823117 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.822448 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.826341 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.825769 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g2nv5\"" Jul 06 15:28:46.826341 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.825919 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 15:28:46.826341 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.825998 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 15:28:46.826341 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.826150 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 15:28:46.874455 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.874373 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:46.881793 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.881717 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-accelerators-collector-config\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.881793 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.881770 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-tls\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.882020 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.881802 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c3474713-76cd-441b-97a4-d7244637c9fb-root\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.882020 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.881837 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb8hd\" (UniqueName: \"kubernetes.io/projected/c3474713-76cd-441b-97a4-d7244637c9fb-kube-api-access-vb8hd\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.882020 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.881898 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.882020 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.881927 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c3474713-76cd-441b-97a4-d7244637c9fb-sys\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.882020 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.881963 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-wtmp\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.882020 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.881991 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-textfile\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.882370 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.882027 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3474713-76cd-441b-97a4-d7244637c9fb-metrics-client-ca\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.941657 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.941601 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:46.991032 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.983829 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb8hd\" (UniqueName: \"kubernetes.io/projected/c3474713-76cd-441b-97a4-d7244637c9fb-kube-api-access-vb8hd\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.991032 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.983897 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.991032 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.984072 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c3474713-76cd-441b-97a4-d7244637c9fb-sys\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.991032 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.984118 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-wtmp\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.991032 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.984155 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-textfile\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.991032 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.984366 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3474713-76cd-441b-97a4-d7244637c9fb-metrics-client-ca\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.991032 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.984436 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-accelerators-collector-config\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.991032 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.984478 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-tls\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.991032 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.984688 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c3474713-76cd-441b-97a4-d7244637c9fb-root\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.991032 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.990242 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c3474713-76cd-441b-97a4-d7244637c9fb-sys\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.993547 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.992125 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3474713-76cd-441b-97a4-d7244637c9fb-metrics-client-ca\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.993547 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.992267 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-wtmp\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.993547 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.992501 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-textfile\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.993547 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.992955 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-accelerators-collector-config\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:46.993547 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:46.993066 2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 15:28:46.993547 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:46.993130 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-tls podName:c3474713-76cd-441b-97a4-d7244637c9fb nodeName:}" failed. No retries permitted until 2026-07-06 15:28:47.493111669 +0000 UTC m=+149.125663646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-tls") pod "node-exporter-5wrkh" (UID: "c3474713-76cd-441b-97a4-d7244637c9fb") : secret "node-exporter-tls" not found Jul 06 15:28:46.995550 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:46.993927 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c3474713-76cd-441b-97a4-d7244637c9fb-root\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:47.004947 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:47.004926 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb8hd\" (UniqueName: \"kubernetes.io/projected/c3474713-76cd-441b-97a4-d7244637c9fb-kube-api-access-vb8hd\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:47.012196 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:47.012174 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:47.110159 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:47.110032 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:47.237497 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:47.237372 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:47.333840 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:47.333672 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:47.590824 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:47.590777 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-tls\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:47.591178 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:47.590894 2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 15:28:47.591178 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:47.590961 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-tls podName:c3474713-76cd-441b-97a4-d7244637c9fb nodeName:}" failed. No retries permitted until 2026-07-06 15:28:48.590944006 +0000 UTC m=+150.223495970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-tls") pod "node-exporter-5wrkh" (UID: "c3474713-76cd-441b-97a4-d7244637c9fb") : secret "node-exporter-tls" not found Jul 06 15:28:47.628565 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:47.627791 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:47.729549 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:47.728773 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:47.828312 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:47.828247 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:47.914508 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:47.914350 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:48.159073 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:48.159029 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:48.344800 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:48.344755 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:48.432864 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:48.432826 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:48.602002 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:48.601916 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-tls\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:48.604866 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:48.604831 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c3474713-76cd-441b-97a4-d7244637c9fb-node-exporter-tls\") pod \"node-exporter-5wrkh\" (UID: \"c3474713-76cd-441b-97a4-d7244637c9fb\") " pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:48.638442 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:48.638412 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5wrkh" Jul 06 15:28:49.826318 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:49.826157 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:49.948036 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:49.947934 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:50.052872 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:50.052765 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:50.241209 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:50.241158 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:51.640253 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.640194 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:51.680215 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.680181 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9"] Jul 06 15:28:51.683404 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.683382 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9" Jul 06 15:28:51.686686 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.686663 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 15:28:51.686788 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.686773 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-g2tgv\"" Jul 06 15:28:51.694792 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.693477 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9"] Jul 06 15:28:51.730650 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.730619 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5369321f-60a5-4cd6-bdad-9ae4e17f436d-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-np2l9\" (UID: \"5369321f-60a5-4cd6-bdad-9ae4e17f436d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9" Jul 06 15:28:51.730803 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.730703 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdmgq\" (UniqueName: \"kubernetes.io/projected/5369321f-60a5-4cd6-bdad-9ae4e17f436d-kube-api-access-fdmgq\") pod \"monitoring-plugin-d5dcff7cf-np2l9\" (UID: \"5369321f-60a5-4cd6-bdad-9ae4e17f436d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9" Jul 06 15:28:51.735099 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.735052 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:51.821989 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.821939 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:51.831774 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.831743 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5369321f-60a5-4cd6-bdad-9ae4e17f436d-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-np2l9\" (UID: \"5369321f-60a5-4cd6-bdad-9ae4e17f436d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9" Jul 06 15:28:51.831903 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.831822 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdmgq\" (UniqueName: \"kubernetes.io/projected/5369321f-60a5-4cd6-bdad-9ae4e17f436d-kube-api-access-fdmgq\") pod \"monitoring-plugin-d5dcff7cf-np2l9\" (UID: \"5369321f-60a5-4cd6-bdad-9ae4e17f436d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9" Jul 06 15:28:51.834609 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.834587 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5369321f-60a5-4cd6-bdad-9ae4e17f436d-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-np2l9\" (UID: \"5369321f-60a5-4cd6-bdad-9ae4e17f436d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9" Jul 06 15:28:51.842265 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.842241 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdmgq\" (UniqueName: \"kubernetes.io/projected/5369321f-60a5-4cd6-bdad-9ae4e17f436d-kube-api-access-fdmgq\") pod \"monitoring-plugin-d5dcff7cf-np2l9\" (UID: \"5369321f-60a5-4cd6-bdad-9ae4e17f436d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9" Jul 06 15:28:51.905186 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.905104 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:51.998020 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:51.997980 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9" Jul 06 15:28:52.074034 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.074003 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-696bd4df5b-9xbj7"] Jul 06 15:28:52.079585 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.078131 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.084000 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.083977 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 15:28:52.084141 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.084024 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 15:28:52.084141 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.084038 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 15:28:52.084299 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.084274 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 15:28:52.084299 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.084296 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ss9xj\"" Jul 06 15:28:52.084473 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.084310 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 15:28:52.090226 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.090201 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 15:28:52.092491 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.092471 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-696bd4df5b-9xbj7"] Jul 06 15:28:52.135416 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.135378 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5e8b26-1ec1-419e-af70-3d9947dd2c52-metrics-client-ca\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.135416 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.135415 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5n6t\" (UniqueName: \"kubernetes.io/projected/da5e8b26-1ec1-419e-af70-3d9947dd2c52-kube-api-access-r5n6t\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.135642 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.135440 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5e8b26-1ec1-419e-af70-3d9947dd2c52-serving-certs-ca-bundle\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.135642 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.135556 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/da5e8b26-1ec1-419e-af70-3d9947dd2c52-secret-telemeter-client\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.135642 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.135591 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/da5e8b26-1ec1-419e-af70-3d9947dd2c52-federate-client-tls\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.135794 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.135651 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/da5e8b26-1ec1-419e-af70-3d9947dd2c52-telemeter-client-tls\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.135794 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.135682 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da5e8b26-1ec1-419e-af70-3d9947dd2c52-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.135794 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.135725 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5e8b26-1ec1-419e-af70-3d9947dd2c52-telemeter-trusted-ca-bundle\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.236765 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.236725 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5e8b26-1ec1-419e-af70-3d9947dd2c52-metrics-client-ca\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.236943 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.236777 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5n6t\" (UniqueName: \"kubernetes.io/projected/da5e8b26-1ec1-419e-af70-3d9947dd2c52-kube-api-access-r5n6t\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.236943 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.236802 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5e8b26-1ec1-419e-af70-3d9947dd2c52-serving-certs-ca-bundle\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.236943 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.236823 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/da5e8b26-1ec1-419e-af70-3d9947dd2c52-secret-telemeter-client\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.236943 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.236846 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/da5e8b26-1ec1-419e-af70-3d9947dd2c52-federate-client-tls\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.236943 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.236903 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/da5e8b26-1ec1-419e-af70-3d9947dd2c52-telemeter-client-tls\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.236943 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.236936 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da5e8b26-1ec1-419e-af70-3d9947dd2c52-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.237305 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.237282 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5e8b26-1ec1-419e-af70-3d9947dd2c52-telemeter-trusted-ca-bundle\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.237555 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.237509 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da5e8b26-1ec1-419e-af70-3d9947dd2c52-metrics-client-ca\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.237935 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.237868 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5e8b26-1ec1-419e-af70-3d9947dd2c52-serving-certs-ca-bundle\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.238211 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.238163 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5e8b26-1ec1-419e-af70-3d9947dd2c52-telemeter-trusted-ca-bundle\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.240220 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.240193 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/da5e8b26-1ec1-419e-af70-3d9947dd2c52-federate-client-tls\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.240380 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.240359 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da5e8b26-1ec1-419e-af70-3d9947dd2c52-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.240629 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.240610 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/da5e8b26-1ec1-419e-af70-3d9947dd2c52-telemeter-client-tls\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.241004 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.240981 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/da5e8b26-1ec1-419e-af70-3d9947dd2c52-secret-telemeter-client\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.246225 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.246205 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5n6t\" (UniqueName: \"kubernetes.io/projected/da5e8b26-1ec1-419e-af70-3d9947dd2c52-kube-api-access-r5n6t\") pod \"telemeter-client-696bd4df5b-9xbj7\" (UID: \"da5e8b26-1ec1-419e-af70-3d9947dd2c52\") " pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.392375 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.392345 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" Jul 06 15:28:52.660214 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.660126 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6478fb58db-d5xmd"] Jul 06 15:28:52.663351 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.663325 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.671379 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.671346 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 15:28:52.676213 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.676192 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6478fb58db-d5xmd"] Jul 06 15:28:52.742482 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.742451 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-trusted-ca-bundle\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.742666 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.742491 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2847c482-6468-4427-9cc2-61ebdb91ed68-console-serving-cert\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.742666 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.742521 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8gcn\" (UniqueName: \"kubernetes.io/projected/2847c482-6468-4427-9cc2-61ebdb91ed68-kube-api-access-w8gcn\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.742666 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.742650 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-console-config\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.742840 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.742725 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2847c482-6468-4427-9cc2-61ebdb91ed68-console-oauth-config\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.742840 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.742746 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-oauth-serving-cert\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.742840 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.742773 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-service-ca\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.843928 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.843882 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-console-config\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.844088 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.843945 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2847c482-6468-4427-9cc2-61ebdb91ed68-console-oauth-config\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.844088 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.843974 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-oauth-serving-cert\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.844088 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.844008 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-service-ca\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.844088 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.844068 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-trusted-ca-bundle\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.844309 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.844097 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2847c482-6468-4427-9cc2-61ebdb91ed68-console-serving-cert\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.844309 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.844118 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8gcn\" (UniqueName: \"kubernetes.io/projected/2847c482-6468-4427-9cc2-61ebdb91ed68-kube-api-access-w8gcn\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.844697 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.844671 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-console-config\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.845152 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.845127 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-oauth-serving-cert\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.845702 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.845679 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-service-ca\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.846129 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.846105 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-trusted-ca-bundle\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.847102 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.847077 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2847c482-6468-4427-9cc2-61ebdb91ed68-console-serving-cert\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.847473 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.847453 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2847c482-6468-4427-9cc2-61ebdb91ed68-console-oauth-config\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.855113 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.855078 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8gcn\" (UniqueName: \"kubernetes.io/projected/2847c482-6468-4427-9cc2-61ebdb91ed68-kube-api-access-w8gcn\") pod \"console-6478fb58db-d5xmd\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:52.977721 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:52.975338 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:28:53.145973 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.145937 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:28:53.149295 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.149263 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:53.150061 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.150032 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.155072 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.154992 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 15:28:53.155233 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.155186 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 15:28:53.155322 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.155270 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 15:28:53.155700 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.155512 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-h8ggc\"" Jul 06 15:28:53.155700 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.155517 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 15:28:53.157545 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.157495 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 15:28:53.157965 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.157947 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 15:28:53.158119 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.158098 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 15:28:53.158188 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.158151 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 15:28:53.158374 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.158356 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 15:28:53.158520 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.158493 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bkgu8e903q43u\"" Jul 06 15:28:53.158520 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.158513 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 15:28:53.158704 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.158637 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 15:28:53.158778 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.158761 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 15:28:53.158849 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.158772 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 15:28:53.158849 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.158821 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 15:28:53.160311 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.160290 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 15:28:53.180728 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.180697 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:28:53.249691 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.249614 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.249691 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.249669 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-config-out\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.249923 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.249698 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.249923 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.249786 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.249923 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.249824 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.249923 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.249847 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.249923 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.249895 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.250164 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.249944 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.250164 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.249980 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.250164 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.250006 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.250164 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.250027 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.250164 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.250046 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.250164 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.250107 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.250164 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.250124 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.250164 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.250149 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-web-config\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.250522 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.250214 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-config\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.250522 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.250260 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.250522 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.250302 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.250522 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.250323 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.250522 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.250342 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf444\" (UniqueName: \"kubernetes.io/projected/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-kube-api-access-rf444\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.352855 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.352823 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.353041 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.352865 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-config-out\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.353041 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.352909 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.353041 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.352943 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.353041 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.352971 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.353041 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.352997 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.353041 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.353022 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.353327 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.353060 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.353327 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.353095 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.353327 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.353120 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.353327 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.353148 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.354795 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.354392 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.354795 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.354617 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.354795 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.354763 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.355132 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.355104 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.355216 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.355165 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.355216 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.355213 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.355389 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.355240 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.355389 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.355264 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-web-config\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.355389 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.355315 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-config\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.355389 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.355359 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.355627 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.355416 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.355627 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.355432 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.355627 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.355440 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.355627 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.355465 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf444\" (UniqueName: \"kubernetes.io/projected/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-kube-api-access-rf444\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.356645 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.356244 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.356645 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.356349 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-config-out\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.356645 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.356433 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.356645 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.356452 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.357242 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.357135 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.358617 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.358472 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.359341 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.359268 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.359655 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.359617 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-config\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.360177 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.360153 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.360410 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.360384 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.362640 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.360449 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.362640 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.361207 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.362794 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.362695 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-web-config\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.363393 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.363373 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.366870 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.366829 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf444\" (UniqueName: \"kubernetes.io/projected/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-kube-api-access-rf444\") pod \"prometheus-k8s-0\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:53.463846 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:53.463806 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:28:55.128969 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:55.128934 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:55.229422 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:55.229388 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:56.201721 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:56.201683 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3474713_76cd_441b_97a4_d7244637c9fb.slice/crio-cb64ea7911a354521a9d39ad163fb17624c6b9b97f797e05f3c946886de74c2c WatchSource:0}: Error finding container cb64ea7911a354521a9d39ad163fb17624c6b9b97f797e05f3c946886de74c2c: Status 404 returned error can't find the container with id cb64ea7911a354521a9d39ad163fb17624c6b9b97f797e05f3c946886de74c2c Jul 06 15:28:56.380836 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:56.380342 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:28:56.397148 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:56.396298 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-2qq7v" event={"ID":"55501abc-cc50-4838-b8ac-c28284b253f0","Type":"ContainerStarted","Data":"c4332db6e6a44a03125c07eb3dd24379c1b9092f6042ffa43ae9fec4295e6b88"} Jul 06 15:28:56.397970 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:56.397920 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-2qq7v" Jul 06 15:28:56.398839 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:56.398564 2561 patch_prober.go:28] interesting pod/downloads-685bfb5f64-2qq7v container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.16:8080/\": dial tcp 10.133.0.16:8080: connect: connection refused" start-of-body= Jul 06 15:28:56.398839 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:56.398613 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-685bfb5f64-2qq7v" podUID="55501abc-cc50-4838-b8ac-c28284b253f0" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.16:8080/\": dial tcp 10.133.0.16:8080: connect: connection refused" Jul 06 15:28:56.401253 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:56.399597 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-696bd4df5b-9xbj7"] Jul 06 15:28:56.401253 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:56.401143 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6f83a99-4b1c-424d-a6bd-f3f10070b14f","Type":"ContainerStarted","Data":"b03bbef3e512662cc8215b319a6a223136c848f0240b85eae8ae9d9d9799a440"} Jul 06 15:28:56.404561 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:56.404447 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d7bff7f-n5zfn" event={"ID":"d94cdbae-fb8c-404a-bdf4-6fa955bccab9","Type":"ContainerStarted","Data":"a5764b862f3baaaf6f1bebefde1e7c8e9daec2cd9db617deda2ea59ec6e9332f"} Jul 06 15:28:56.404920 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:56.404896 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5e8b26_1ec1_419e_af70_3d9947dd2c52.slice/crio-3838d70dced1b23665e641537dac6cfd08d3419316a691e93b1e9ad6646af01b WatchSource:0}: Error finding container 3838d70dced1b23665e641537dac6cfd08d3419316a691e93b1e9ad6646af01b: Status 404 returned error can't find the container with id 3838d70dced1b23665e641537dac6cfd08d3419316a691e93b1e9ad6646af01b Jul 06 15:28:56.407703 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:56.407682 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5wrkh" event={"ID":"c3474713-76cd-441b-97a4-d7244637c9fb","Type":"ContainerStarted","Data":"cb64ea7911a354521a9d39ad163fb17624c6b9b97f797e05f3c946886de74c2c"} Jul 06 15:28:56.414656 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:56.414432 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9"] Jul 06 15:28:56.420443 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:56.415782 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5369321f_60a5_4cd6_bdad_9ae4e17f436d.slice/crio-3c86a684bc0d574af73d18cbdab7410825d53e313c512239e4415f200d68dbe1 WatchSource:0}: Error finding container 3c86a684bc0d574af73d18cbdab7410825d53e313c512239e4415f200d68dbe1: Status 404 returned error can't find the container with id 3c86a684bc0d574af73d18cbdab7410825d53e313c512239e4415f200d68dbe1 Jul 06 15:28:56.420443 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:56.420130 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-2qq7v" podStartSLOduration=1.033405387 podStartE2EDuration="20.420112306s" totalImagesPullingTime="19.386706919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:56.41895253 +0000 UTC m=+158.051504516" watchObservedRunningTime="2026-07-06 15:28:56.420112306 +0000 UTC m=+158.052664290" Jul 06 15:28:56.439106 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:56.439065 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9d7bff7f-n5zfn" podStartSLOduration=0.988868251 podStartE2EDuration="11.439051393s" totalImagesPullingTime="10.450183142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:56.437550285 +0000 UTC m=+158.070102272" watchObservedRunningTime="2026-07-06 15:28:56.439051393 +0000 UTC m=+158.071603376" Jul 06 15:28:56.442236 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:56.442178 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6478fb58db-d5xmd"] Jul 06 15:28:56.445000 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:28:56.444976 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2847c482_6468_4427_9cc2_61ebdb91ed68.slice/crio-eda9b8324bc9284cdf8a3299e846781e16b18c856f670c3756440617b5a1be20 WatchSource:0}: Error finding container eda9b8324bc9284cdf8a3299e846781e16b18c856f670c3756440617b5a1be20: Status 404 returned error can't find the container with id eda9b8324bc9284cdf8a3299e846781e16b18c856f670c3756440617b5a1be20 Jul 06 15:28:57.280888 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:28:57.280833 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6bhc8" podUID="cb50cb21-f13c-4d8c-ac8a-088425ccbd50" Jul 06 15:28:57.418573 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:57.418019 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6478fb58db-d5xmd" event={"ID":"2847c482-6468-4427-9cc2-61ebdb91ed68","Type":"ContainerStarted","Data":"3197fc2460843996dba2bf16e523dceb85dff499113d9731fb2576bfc95362e8"} Jul 06 15:28:57.418573 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:57.418083 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6478fb58db-d5xmd" event={"ID":"2847c482-6468-4427-9cc2-61ebdb91ed68","Type":"ContainerStarted","Data":"eda9b8324bc9284cdf8a3299e846781e16b18c856f670c3756440617b5a1be20"} Jul 06 15:28:57.421459 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:57.421369 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" event={"ID":"da5e8b26-1ec1-419e-af70-3d9947dd2c52","Type":"ContainerStarted","Data":"3838d70dced1b23665e641537dac6cfd08d3419316a691e93b1e9ad6646af01b"} Jul 06 15:28:57.423790 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:57.423586 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6bhc8" Jul 06 15:28:57.424695 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:57.424633 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9" event={"ID":"5369321f-60a5-4cd6-bdad-9ae4e17f436d","Type":"ContainerStarted","Data":"3c86a684bc0d574af73d18cbdab7410825d53e313c512239e4415f200d68dbe1"} Jul 06 15:28:57.427067 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:57.426941 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-2qq7v" Jul 06 15:28:57.446557 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:57.446077 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6478fb58db-d5xmd" podStartSLOduration=5.446058514 podStartE2EDuration="5.446058514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:57.443373393 +0000 UTC m=+159.075925376" watchObservedRunningTime="2026-07-06 15:28:57.446058514 +0000 UTC m=+159.078610499" Jul 06 15:28:57.887691 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:57.887573 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:57.958058 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:57.958030 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:58.321792 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:58.321764 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c6fd88d46-wfqsl" Jul 06 15:28:58.435432 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:58.435346 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5wrkh" event={"ID":"c3474713-76cd-441b-97a4-d7244637c9fb","Type":"ContainerStarted","Data":"6f908ed501301795e6de4c3e874494f662366241d16f70bece3530a3c881e658"} Jul 06 15:28:58.496749 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:58.496670 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:58.581200 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:58.581078 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:58.658236 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:58.658198 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:58.951819 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:58.951733 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:59.064725 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:59.064667 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:59.193684 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:59.193599 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:28:59.262064 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:28:59.262028 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:00.446576 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:00.446522 2561 generic.go:356] "Generic (PLEG): container finished" podID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerID="a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919" exitCode=0 Jul 06 15:29:00.446963 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:00.446567 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6f83a99-4b1c-424d-a6bd-f3f10070b14f","Type":"ContainerDied","Data":"a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919"} Jul 06 15:29:00.448978 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:00.448953 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" event={"ID":"da5e8b26-1ec1-419e-af70-3d9947dd2c52","Type":"ContainerStarted","Data":"a0def0dc3065f86d490b0cf85525d087bacd6afd98ed0e9343c835ca572b41a0"} Jul 06 15:29:00.449073 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:00.449016 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" event={"ID":"da5e8b26-1ec1-419e-af70-3d9947dd2c52","Type":"ContainerStarted","Data":"0514a06357a6e53f996d180fb58b854c2adf8db9e0cc98bac3d12fa5ec32070c"} Jul 06 15:29:00.451591 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:00.451562 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9" event={"ID":"5369321f-60a5-4cd6-bdad-9ae4e17f436d","Type":"ContainerStarted","Data":"4d70a7260b7edce44acaf2bb3bbcb39af7f8c6fdf114880cc4a42e3df48ccbcb"} Jul 06 15:29:00.452144 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:00.452082 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9" Jul 06 15:29:00.457340 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:00.456692 2561 generic.go:356] "Generic (PLEG): container finished" podID="c3474713-76cd-441b-97a4-d7244637c9fb" containerID="6f908ed501301795e6de4c3e874494f662366241d16f70bece3530a3c881e658" exitCode=0 Jul 06 15:29:00.457340 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:00.456727 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5wrkh" event={"ID":"c3474713-76cd-441b-97a4-d7244637c9fb","Type":"ContainerDied","Data":"6f908ed501301795e6de4c3e874494f662366241d16f70bece3530a3c881e658"} Jul 06 15:29:00.461913 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:00.461889 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9" Jul 06 15:29:00.515880 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:00.515831 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-np2l9" podStartSLOduration=5.898153178 podStartE2EDuration="9.515815302s" totalImagesPullingTime="3.617662124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:29:00.514403441 +0000 UTC m=+162.146955425" watchObservedRunningTime="2026-07-06 15:29:00.515815302 +0000 UTC m=+162.148367284" Jul 06 15:29:00.827817 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:00.827735 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:00.930581 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:00.930539 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:01.465824 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:01.464845 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" event={"ID":"da5e8b26-1ec1-419e-af70-3d9947dd2c52","Type":"ContainerStarted","Data":"77811783e8ea067f8847a1483ce242c10478be38182cb7ea984d9e910bf70c4f"} Jul 06 15:29:01.468893 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:01.468858 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5wrkh" event={"ID":"c3474713-76cd-441b-97a4-d7244637c9fb","Type":"ContainerStarted","Data":"ca6e07d5e49f3d2c4884fe521c8e0651bf49958a71f8a5e53256c1580b2edcaa"} Jul 06 15:29:01.469044 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:01.468918 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5wrkh" event={"ID":"c3474713-76cd-441b-97a4-d7244637c9fb","Type":"ContainerStarted","Data":"7723d25ec325d2895a725c49a0146251eca60c6926c56718552762e301d50502"} Jul 06 15:29:01.493505 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:01.492212 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-696bd4df5b-9xbj7" podStartSLOduration=5.855073309 podStartE2EDuration="9.492191228s" totalImagesPullingTime="3.637117919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:29:01.490749703 +0000 UTC m=+163.123301686" watchObservedRunningTime="2026-07-06 15:29:01.492191228 +0000 UTC m=+163.124743213" Jul 06 15:29:01.512014 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:01.511961 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5wrkh" podStartSLOduration=14.068409921 podStartE2EDuration="15.511938927s" totalImagesPullingTime="1.33167094s" totalInitContainerRuntime="111.858066ms" isStatefulPod=true podCreationTimestamp="2026-07-06 15:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:29:01.510976801 +0000 UTC m=+163.143528805" watchObservedRunningTime="2026-07-06 15:29:01.511938927 +0000 UTC m=+163.144490914" Jul 06 15:29:02.159266 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:02.159227 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:29:02.163283 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:02.163227 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb50cb21-f13c-4d8c-ac8a-088425ccbd50-metrics-tls\") pod \"dns-default-6bhc8\" (UID: \"cb50cb21-f13c-4d8c-ac8a-088425ccbd50\") " pod="openshift-dns/dns-default-6bhc8" Jul 06 15:29:02.195169 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:02.195135 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9d7bff7f-n5zfn"] Jul 06 15:29:02.239500 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:02.234637 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cwwg8\"" Jul 06 15:29:02.242395 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:02.241817 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6bhc8" Jul 06 15:29:02.471811 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:02.471648 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6bhc8"] Jul 06 15:29:02.578555 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:02.576094 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:02.676728 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:02.676643 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:02.976333 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:02.976282 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:29:02.976728 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:02.976704 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:29:02.982768 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:02.982745 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:29:03.302568 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:03.302424 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:03.376102 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:03.376067 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:03.492093 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:03.492003 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6bhc8" event={"ID":"cb50cb21-f13c-4d8c-ac8a-088425ccbd50","Type":"ContainerStarted","Data":"33b9a9f0b6bca18c1593513dfc345cdcde52d7023ffdd66f1f6254d4e92e081a"} Jul 06 15:29:03.498559 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:03.498417 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:29:04.319930 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:04.319896 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:04.410895 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:04.410788 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:05.594136 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:05.594109 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:29:06.507269 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:06.507237 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6f83a99-4b1c-424d-a6bd-f3f10070b14f","Type":"ContainerStarted","Data":"f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f"} Jul 06 15:29:06.507479 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:06.507352 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6f83a99-4b1c-424d-a6bd-f3f10070b14f","Type":"ContainerStarted","Data":"d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a"} Jul 06 15:29:06.508979 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:06.508955 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6bhc8" event={"ID":"cb50cb21-f13c-4d8c-ac8a-088425ccbd50","Type":"ContainerStarted","Data":"60f60e40f23402986cbca83da817cdcdeca887d43e49fb5c6d57c151d7d27a32"} Jul 06 15:29:06.509097 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:06.508995 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6bhc8" event={"ID":"cb50cb21-f13c-4d8c-ac8a-088425ccbd50","Type":"ContainerStarted","Data":"07c28b47227a16b4213c503a7b3dc00ba34c0deb5dc0e5c28652ca96d79c14b6"} Jul 06 15:29:06.509097 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:06.509085 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6bhc8" Jul 06 15:29:06.533710 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:06.533668 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6bhc8" podStartSLOduration=129.25408303 podStartE2EDuration="2m12.533654383s" totalImagesPullingTime="3.279571353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:29:06.532163018 +0000 UTC m=+168.164715002" watchObservedRunningTime="2026-07-06 15:29:06.533654383 +0000 UTC m=+168.166206366" Jul 06 15:29:06.796481 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:06.796403 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:06.873940 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:06.873905 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:08.153517 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:08.153481 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:08.226411 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:08.226378 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:09.527025 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:09.526956 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6f83a99-4b1c-424d-a6bd-f3f10070b14f","Type":"ContainerStarted","Data":"3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4"} Jul 06 15:29:09.527025 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:09.527013 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6f83a99-4b1c-424d-a6bd-f3f10070b14f","Type":"ContainerStarted","Data":"4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318"} Jul 06 15:29:09.527466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:09.527050 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6f83a99-4b1c-424d-a6bd-f3f10070b14f","Type":"ContainerStarted","Data":"093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9"} Jul 06 15:29:09.527466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:09.527075 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6f83a99-4b1c-424d-a6bd-f3f10070b14f","Type":"ContainerStarted","Data":"758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22"} Jul 06 15:29:09.573476 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:09.573425 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.125452274 podStartE2EDuration="16.573403868s" totalImagesPullingTime="11.414871278s" totalInitContainerRuntime="33.080316ms" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:29:09.570383468 +0000 UTC m=+171.202935451" watchObservedRunningTime="2026-07-06 15:29:09.573403868 +0000 UTC m=+171.205955853" Jul 06 15:29:10.029807 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:10.029769 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:10.135850 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:10.135814 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:12.813985 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:12.813956 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6478fb58db-d5xmd"] Jul 06 15:29:12.926951 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:12.926918 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:13.032367 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:13.032342 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:13.130677 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:13.130612 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:13.464666 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:13.464599 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:29:13.955791 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:13.955754 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:14.026236 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:14.026199 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:15.553321 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:15.553287 2561 generic.go:356] "Generic (PLEG): container finished" podID="48ef32e0-be01-49ac-ae54-32f33476eadc" containerID="5b8d83fd45bced1b8b97cf539ec7429cf69ddcc3a0ea79bc9886e5a3b2941a38" exitCode=0 Jul 06 15:29:15.553786 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:15.553360 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" event={"ID":"48ef32e0-be01-49ac-ae54-32f33476eadc","Type":"ContainerDied","Data":"5b8d83fd45bced1b8b97cf539ec7429cf69ddcc3a0ea79bc9886e5a3b2941a38"} Jul 06 15:29:15.553786 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:15.553757 2561 scope.go:117] "RemoveContainer" containerID="5b8d83fd45bced1b8b97cf539ec7429cf69ddcc3a0ea79bc9886e5a3b2941a38" Jul 06 15:29:16.516305 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:16.516277 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6bhc8" Jul 06 15:29:16.559656 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:16.559626 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xmrmx" event={"ID":"48ef32e0-be01-49ac-ae54-32f33476eadc","Type":"ContainerStarted","Data":"95a71f078467bc8c1ff263bf6da19e1a32fbb8f255aaabff80dcb842a38c7dd0"} Jul 06 15:29:21.233921 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:21.233887 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:21.334266 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:21.334228 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:21.403248 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:21.403211 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:25.596253 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:25.596220 2561 generic.go:356] "Generic (PLEG): container finished" podID="705cc873-6a42-42ad-8bdb-dbc28545a42f" containerID="5d00d316ad5ad75ff1daa6c419b33b80fb147a4a6a4f546a523de14bdefc7b60" exitCode=0 Jul 06 15:29:25.596659 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:25.596295 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" event={"ID":"705cc873-6a42-42ad-8bdb-dbc28545a42f","Type":"ContainerDied","Data":"5d00d316ad5ad75ff1daa6c419b33b80fb147a4a6a4f546a523de14bdefc7b60"} Jul 06 15:29:25.596659 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:25.596620 2561 scope.go:117] "RemoveContainer" containerID="5d00d316ad5ad75ff1daa6c419b33b80fb147a4a6a4f546a523de14bdefc7b60" Jul 06 15:29:25.675352 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:25.675108 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:25.752970 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:25.752939 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:26.602008 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:26.601972 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-f5b2g" event={"ID":"705cc873-6a42-42ad-8bdb-dbc28545a42f","Type":"ContainerStarted","Data":"ab3bb7794cdd0512719cd41fe86fee8f0fe52f5643b1d4954d7d0b69b6e7c223"} Jul 06 15:29:27.230917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.230885 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-9d7bff7f-n5zfn" podUID="d94cdbae-fb8c-404a-bdf4-6fa955bccab9" containerName="console" containerID="cri-o://a5764b862f3baaaf6f1bebefde1e7c8e9daec2cd9db617deda2ea59ec6e9332f" gracePeriod=15 Jul 06 15:29:27.481972 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.481920 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:29:27.575759 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.575731 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-config\") pod \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " Jul 06 15:29:27.575759 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.575757 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-service-ca\") pod \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " Jul 06 15:29:27.575948 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.575795 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-oauth-config\") pod \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " Jul 06 15:29:27.575948 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.575826 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn8s9\" (UniqueName: \"kubernetes.io/projected/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-kube-api-access-kn8s9\") pod \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " Jul 06 15:29:27.575948 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.575850 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-oauth-serving-cert\") pod \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " Jul 06 15:29:27.575948 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.575887 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-serving-cert\") pod \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\" (UID: \"d94cdbae-fb8c-404a-bdf4-6fa955bccab9\") " Jul 06 15:29:27.576133 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.576118 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-config" (OuterVolumeSpecName: "console-config") pod "d94cdbae-fb8c-404a-bdf4-6fa955bccab9" (UID: "d94cdbae-fb8c-404a-bdf4-6fa955bccab9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:29:27.576193 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.576157 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-service-ca" (OuterVolumeSpecName: "service-ca") pod "d94cdbae-fb8c-404a-bdf4-6fa955bccab9" (UID: "d94cdbae-fb8c-404a-bdf4-6fa955bccab9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:29:27.576293 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.576267 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d94cdbae-fb8c-404a-bdf4-6fa955bccab9" (UID: "d94cdbae-fb8c-404a-bdf4-6fa955bccab9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:29:27.578160 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.578134 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d94cdbae-fb8c-404a-bdf4-6fa955bccab9" (UID: "d94cdbae-fb8c-404a-bdf4-6fa955bccab9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:29:27.578261 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.578189 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-kube-api-access-kn8s9" (OuterVolumeSpecName: "kube-api-access-kn8s9") pod "d94cdbae-fb8c-404a-bdf4-6fa955bccab9" (UID: "d94cdbae-fb8c-404a-bdf4-6fa955bccab9"). InnerVolumeSpecName "kube-api-access-kn8s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:29:27.578261 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.578221 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d94cdbae-fb8c-404a-bdf4-6fa955bccab9" (UID: "d94cdbae-fb8c-404a-bdf4-6fa955bccab9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:29:27.606920 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.606892 2561 generic.go:356] "Generic (PLEG): container finished" podID="d94cdbae-fb8c-404a-bdf4-6fa955bccab9" containerID="a5764b862f3baaaf6f1bebefde1e7c8e9daec2cd9db617deda2ea59ec6e9332f" exitCode=0 Jul 06 15:29:27.607210 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.606931 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d7bff7f-n5zfn" Jul 06 15:29:27.607210 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.606974 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d7bff7f-n5zfn" event={"ID":"d94cdbae-fb8c-404a-bdf4-6fa955bccab9","Type":"ContainerDied","Data":"a5764b862f3baaaf6f1bebefde1e7c8e9daec2cd9db617deda2ea59ec6e9332f"} Jul 06 15:29:27.607210 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.607026 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d7bff7f-n5zfn" event={"ID":"d94cdbae-fb8c-404a-bdf4-6fa955bccab9","Type":"ContainerDied","Data":"e6f4c009900f979126d24f6e174105e1562e267c03f6a0f888a820dbde9be737"} Jul 06 15:29:27.607210 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.607049 2561 scope.go:117] "RemoveContainer" containerID="a5764b862f3baaaf6f1bebefde1e7c8e9daec2cd9db617deda2ea59ec6e9332f" Jul 06 15:29:27.617095 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.617066 2561 scope.go:117] "RemoveContainer" containerID="a5764b862f3baaaf6f1bebefde1e7c8e9daec2cd9db617deda2ea59ec6e9332f" Jul 06 15:29:27.617763 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:29:27.617736 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5764b862f3baaaf6f1bebefde1e7c8e9daec2cd9db617deda2ea59ec6e9332f\": container with ID starting with a5764b862f3baaaf6f1bebefde1e7c8e9daec2cd9db617deda2ea59ec6e9332f not found: ID does not exist" containerID="a5764b862f3baaaf6f1bebefde1e7c8e9daec2cd9db617deda2ea59ec6e9332f" Jul 06 15:29:27.617856 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.617770 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5764b862f3baaaf6f1bebefde1e7c8e9daec2cd9db617deda2ea59ec6e9332f"} err="failed to get container status \"a5764b862f3baaaf6f1bebefde1e7c8e9daec2cd9db617deda2ea59ec6e9332f\": rpc error: code = NotFound desc = could not find container \"a5764b862f3baaaf6f1bebefde1e7c8e9daec2cd9db617deda2ea59ec6e9332f\": container with ID starting with a5764b862f3baaaf6f1bebefde1e7c8e9daec2cd9db617deda2ea59ec6e9332f not found: ID does not exist" Jul 06 15:29:27.632162 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.632138 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9d7bff7f-n5zfn"] Jul 06 15:29:27.636023 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.636005 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9d7bff7f-n5zfn"] Jul 06 15:29:27.676657 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.676634 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-oauth-config\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:29:27.676657 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.676654 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn8s9\" (UniqueName: \"kubernetes.io/projected/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-kube-api-access-kn8s9\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:29:27.676772 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.676666 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-oauth-serving-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:29:27.676772 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.676674 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-serving-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:29:27.676772 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.676682 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-console-config\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:29:27.676772 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:27.676690 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d94cdbae-fb8c-404a-bdf4-6fa955bccab9-service-ca\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:29:28.798739 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:28.798712 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d94cdbae-fb8c-404a-bdf4-6fa955bccab9" path="/var/lib/kubelet/pods/d94cdbae-fb8c-404a-bdf4-6fa955bccab9/volumes" Jul 06 15:29:28.954012 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:28.953975 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:29.056289 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:29.056192 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:29:37.841895 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:37.841832 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6478fb58db-d5xmd" podUID="2847c482-6468-4427-9cc2-61ebdb91ed68" containerName="console" containerID="cri-o://3197fc2460843996dba2bf16e523dceb85dff499113d9731fb2576bfc95362e8" gracePeriod=15 Jul 06 15:29:38.122213 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.122186 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:29:38.162978 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.162952 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2847c482-6468-4427-9cc2-61ebdb91ed68-console-oauth-config\") pod \"2847c482-6468-4427-9cc2-61ebdb91ed68\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " Jul 06 15:29:38.163117 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.162985 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-oauth-serving-cert\") pod \"2847c482-6468-4427-9cc2-61ebdb91ed68\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " Jul 06 15:29:38.163227 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.163203 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2847c482-6468-4427-9cc2-61ebdb91ed68-console-serving-cert\") pod \"2847c482-6468-4427-9cc2-61ebdb91ed68\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " Jul 06 15:29:38.163309 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.163265 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-console-config\") pod \"2847c482-6468-4427-9cc2-61ebdb91ed68\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " Jul 06 15:29:38.163309 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.163303 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-trusted-ca-bundle\") pod \"2847c482-6468-4427-9cc2-61ebdb91ed68\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " Jul 06 15:29:38.163416 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.163355 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8gcn\" (UniqueName: \"kubernetes.io/projected/2847c482-6468-4427-9cc2-61ebdb91ed68-kube-api-access-w8gcn\") pod \"2847c482-6468-4427-9cc2-61ebdb91ed68\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " Jul 06 15:29:38.163416 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.163375 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2847c482-6468-4427-9cc2-61ebdb91ed68" (UID: "2847c482-6468-4427-9cc2-61ebdb91ed68"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:29:38.163416 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.163387 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-service-ca\") pod \"2847c482-6468-4427-9cc2-61ebdb91ed68\" (UID: \"2847c482-6468-4427-9cc2-61ebdb91ed68\") " Jul 06 15:29:38.163684 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.163655 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-console-config" (OuterVolumeSpecName: "console-config") pod "2847c482-6468-4427-9cc2-61ebdb91ed68" (UID: "2847c482-6468-4427-9cc2-61ebdb91ed68"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:29:38.163778 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.163730 2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-oauth-serving-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:29:38.163818 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.163768 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2847c482-6468-4427-9cc2-61ebdb91ed68" (UID: "2847c482-6468-4427-9cc2-61ebdb91ed68"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:29:38.164122 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.164094 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-service-ca" (OuterVolumeSpecName: "service-ca") pod "2847c482-6468-4427-9cc2-61ebdb91ed68" (UID: "2847c482-6468-4427-9cc2-61ebdb91ed68"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:29:38.165466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.165438 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2847c482-6468-4427-9cc2-61ebdb91ed68-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2847c482-6468-4427-9cc2-61ebdb91ed68" (UID: "2847c482-6468-4427-9cc2-61ebdb91ed68"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:29:38.165764 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.165736 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2847c482-6468-4427-9cc2-61ebdb91ed68-kube-api-access-w8gcn" (OuterVolumeSpecName: "kube-api-access-w8gcn") pod "2847c482-6468-4427-9cc2-61ebdb91ed68" (UID: "2847c482-6468-4427-9cc2-61ebdb91ed68"). InnerVolumeSpecName "kube-api-access-w8gcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:29:38.166074 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.166049 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2847c482-6468-4427-9cc2-61ebdb91ed68-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2847c482-6468-4427-9cc2-61ebdb91ed68" (UID: "2847c482-6468-4427-9cc2-61ebdb91ed68"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:29:38.264842 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.264819 2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2847c482-6468-4427-9cc2-61ebdb91ed68-console-serving-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:29:38.264842 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.264839 2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-console-config\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:29:38.264975 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.264848 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-trusted-ca-bundle\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:29:38.264975 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.264859 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8gcn\" (UniqueName: \"kubernetes.io/projected/2847c482-6468-4427-9cc2-61ebdb91ed68-kube-api-access-w8gcn\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:29:38.264975 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.264867 2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2847c482-6468-4427-9cc2-61ebdb91ed68-service-ca\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:29:38.264975 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.264876 2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2847c482-6468-4427-9cc2-61ebdb91ed68-console-oauth-config\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:29:38.657103 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.657075 2561 generic.go:356] "Generic (PLEG): container finished" podID="2847c482-6468-4427-9cc2-61ebdb91ed68" containerID="3197fc2460843996dba2bf16e523dceb85dff499113d9731fb2576bfc95362e8" exitCode=0 Jul 06 15:29:38.657240 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.657118 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6478fb58db-d5xmd" Jul 06 15:29:38.657240 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.657140 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6478fb58db-d5xmd" event={"ID":"2847c482-6468-4427-9cc2-61ebdb91ed68","Type":"ContainerDied","Data":"3197fc2460843996dba2bf16e523dceb85dff499113d9731fb2576bfc95362e8"} Jul 06 15:29:38.657240 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.657178 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6478fb58db-d5xmd" event={"ID":"2847c482-6468-4427-9cc2-61ebdb91ed68","Type":"ContainerDied","Data":"eda9b8324bc9284cdf8a3299e846781e16b18c856f670c3756440617b5a1be20"} Jul 06 15:29:38.657240 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.657198 2561 scope.go:117] "RemoveContainer" containerID="3197fc2460843996dba2bf16e523dceb85dff499113d9731fb2576bfc95362e8" Jul 06 15:29:38.665970 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.665955 2561 scope.go:117] "RemoveContainer" containerID="3197fc2460843996dba2bf16e523dceb85dff499113d9731fb2576bfc95362e8" Jul 06 15:29:38.666192 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:29:38.666170 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3197fc2460843996dba2bf16e523dceb85dff499113d9731fb2576bfc95362e8\": container with ID starting with 3197fc2460843996dba2bf16e523dceb85dff499113d9731fb2576bfc95362e8 not found: ID does not exist" containerID="3197fc2460843996dba2bf16e523dceb85dff499113d9731fb2576bfc95362e8" Jul 06 15:29:38.666257 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.666204 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3197fc2460843996dba2bf16e523dceb85dff499113d9731fb2576bfc95362e8"} err="failed to get container status \"3197fc2460843996dba2bf16e523dceb85dff499113d9731fb2576bfc95362e8\": rpc error: code = NotFound desc = could not find container \"3197fc2460843996dba2bf16e523dceb85dff499113d9731fb2576bfc95362e8\": container with ID starting with 3197fc2460843996dba2bf16e523dceb85dff499113d9731fb2576bfc95362e8 not found: ID does not exist" Jul 06 15:29:38.678880 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.678855 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6478fb58db-d5xmd"] Jul 06 15:29:38.683014 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.682994 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6478fb58db-d5xmd"] Jul 06 15:29:38.799762 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:38.799728 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2847c482-6468-4427-9cc2-61ebdb91ed68" path="/var/lib/kubelet/pods/2847c482-6468-4427-9cc2-61ebdb91ed68/volumes" Jul 06 15:29:40.668159 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:40.668129 2561 generic.go:356] "Generic (PLEG): container finished" podID="f6dbf566-cef0-4ead-a469-1a197bce5b3d" containerID="416f5ac8b2f88f2a7888ef55f2a5f625c116498ee040390789ed4c7c11b85428" exitCode=0 Jul 06 15:29:40.668503 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:40.668209 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" event={"ID":"f6dbf566-cef0-4ead-a469-1a197bce5b3d","Type":"ContainerDied","Data":"416f5ac8b2f88f2a7888ef55f2a5f625c116498ee040390789ed4c7c11b85428"} Jul 06 15:29:40.668583 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:40.668569 2561 scope.go:117] "RemoveContainer" containerID="416f5ac8b2f88f2a7888ef55f2a5f625c116498ee040390789ed4c7c11b85428" Jul 06 15:29:41.674321 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:41.674288 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-px9pj" event={"ID":"f6dbf566-cef0-4ead-a469-1a197bce5b3d","Type":"ContainerStarted","Data":"f7b65bf69f1079a7a58daebec6d2debf86121cf6e83c26550a0a46c756e522ea"} Jul 06 15:29:53.464448 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:53.464410 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:29:53.484318 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:53.484288 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:29:53.736004 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:29:53.735979 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:03.018373 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:03.018317 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:03.117578 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:03.117547 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:11.673366 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:11.673329 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:30:11.673882 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:11.673647 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="prometheus" containerID="cri-o://d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a" gracePeriod=600 Jul 06 15:30:11.673882 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:11.673683 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="thanos-sidecar" containerID="cri-o://758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22" gracePeriod=600 Jul 06 15:30:11.673882 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:11.673702 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="kube-rbac-proxy-web" containerID="cri-o://093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9" gracePeriod=600 Jul 06 15:30:11.673882 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:11.673717 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4" gracePeriod=600 Jul 06 15:30:11.673882 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:11.673760 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="config-reloader" containerID="cri-o://f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f" gracePeriod=600 Jul 06 15:30:11.673882 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:11.673663 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="kube-rbac-proxy" containerID="cri-o://4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318" gracePeriod=600 Jul 06 15:30:11.797050 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:11.797026 2561 generic.go:356] "Generic (PLEG): container finished" podID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerID="4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318" exitCode=0 Jul 06 15:30:11.797125 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:11.797052 2561 generic.go:356] "Generic (PLEG): container finished" podID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerID="093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9" exitCode=0 Jul 06 15:30:11.797125 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:11.797058 2561 generic.go:356] "Generic (PLEG): container finished" podID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerID="758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22" exitCode=0 Jul 06 15:30:11.797125 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:11.797082 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6f83a99-4b1c-424d-a6bd-f3f10070b14f","Type":"ContainerDied","Data":"4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318"} Jul 06 15:30:11.797220 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:11.797129 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6f83a99-4b1c-424d-a6bd-f3f10070b14f","Type":"ContainerDied","Data":"093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9"} Jul 06 15:30:11.797220 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:11.797155 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6f83a99-4b1c-424d-a6bd-f3f10070b14f","Type":"ContainerDied","Data":"758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22"} Jul 06 15:30:11.936247 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:11.936221 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:12.030359 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.030330 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-metrics-client-certs\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.030557 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.030370 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-thanos-prometheus-http-client-file\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.030557 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.030398 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-metrics-client-ca\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.030695 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.030518 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-tls-assets\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.030695 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.030598 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.030695 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.030628 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-grpc-tls\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.030695 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.030650 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-web-config\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.030888 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.030702 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf444\" (UniqueName: \"kubernetes.io/projected/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-kube-api-access-rf444\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.030888 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.030743 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-trusted-ca-bundle\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.030888 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.030767 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:30:12.030888 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.030773 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-db\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.031653 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.031110 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-config-out\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.031653 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.031156 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.031653 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.031187 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-serving-certs-ca-bundle\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.031653 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.031248 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-kube-rbac-proxy\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.031653 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.031292 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-2\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.031653 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.031321 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-config\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.031653 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.031361 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-1\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.031653 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.031393 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-tls\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.031653 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.031424 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-kubelet-serving-ca-bundle\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.032587 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.031832 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:30:12.032587 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.031880 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-0\") pod \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\" (UID: \"e6f83a99-4b1c-424d-a6bd-f3f10070b14f\") " Jul 06 15:30:12.032587 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.032059 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:30:12.032587 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.032347 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.032587 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.032368 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-db\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.032587 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.032385 2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-metrics-client-ca\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.033734 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.033025 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:30:12.033734 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.033072 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:30:12.033734 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.033622 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:30:12.034363 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.034096 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:30:12.034363 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.034144 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:30:12.034703 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.034681 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-kube-api-access-rf444" (OuterVolumeSpecName: "kube-api-access-rf444") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "kube-api-access-rf444". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:30:12.034703 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.034681 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:30:12.035134 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.035055 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:30:12.035134 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.035103 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:30:12.035767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.035737 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:30:12.036264 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.036237 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:30:12.036677 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.036631 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:30:12.036888 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.036859 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-config" (OuterVolumeSpecName: "config") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:30:12.036888 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.036881 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:30:12.037309 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.037290 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:30:12.037431 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.037413 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-config-out" (OuterVolumeSpecName: "config-out") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:30:12.046988 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.046971 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-web-config" (OuterVolumeSpecName: "web-config") pod "e6f83a99-4b1c-424d-a6bd-f3f10070b14f" (UID: "e6f83a99-4b1c-424d-a6bd-f3f10070b14f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:30:12.133006 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.132978 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133006 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133003 2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-config\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133014 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133022 2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133032 2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133041 2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133050 2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-metrics-client-certs\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133059 2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133066 2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-tls-assets\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133074 2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133083 2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-grpc-tls\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133090 2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-web-config\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133097 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf444\" (UniqueName: \"kubernetes.io/projected/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-kube-api-access-rf444\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133104 2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-config-out\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133113 2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133121 2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.133146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.133129 2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e6f83a99-4b1c-424d-a6bd-f3f10070b14f-secret-kube-rbac-proxy\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:30:12.808252 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.808224 2561 generic.go:356] "Generic (PLEG): container finished" podID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerID="3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4" exitCode=0 Jul 06 15:30:12.808252 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.808247 2561 generic.go:356] "Generic (PLEG): container finished" podID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerID="f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f" exitCode=0 Jul 06 15:30:12.808252 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.808253 2561 generic.go:356] "Generic (PLEG): container finished" podID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerID="d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a" exitCode=0 Jul 06 15:30:12.808780 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.808298 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:12.808780 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.808298 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6f83a99-4b1c-424d-a6bd-f3f10070b14f","Type":"ContainerDied","Data":"3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4"} Jul 06 15:30:12.808780 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.808391 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6f83a99-4b1c-424d-a6bd-f3f10070b14f","Type":"ContainerDied","Data":"f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f"} Jul 06 15:30:12.808780 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.808447 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6f83a99-4b1c-424d-a6bd-f3f10070b14f","Type":"ContainerDied","Data":"d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a"} Jul 06 15:30:12.808780 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.808464 2561 scope.go:117] "RemoveContainer" containerID="3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4" Jul 06 15:30:12.808780 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.808490 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e6f83a99-4b1c-424d-a6bd-f3f10070b14f","Type":"ContainerDied","Data":"b03bbef3e512662cc8215b319a6a223136c848f0240b85eae8ae9d9d9799a440"} Jul 06 15:30:12.816393 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.816355 2561 scope.go:117] "RemoveContainer" containerID="4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318" Jul 06 15:30:12.823090 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.823076 2561 scope.go:117] "RemoveContainer" containerID="093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9" Jul 06 15:30:12.829668 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.829651 2561 scope.go:117] "RemoveContainer" containerID="758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22" Jul 06 15:30:12.834047 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.834030 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:30:12.837334 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.837315 2561 scope.go:117] "RemoveContainer" containerID="f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f" Jul 06 15:30:12.839677 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.839653 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:30:12.846199 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.846182 2561 scope.go:117] "RemoveContainer" containerID="d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a" Jul 06 15:30:12.853085 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.853067 2561 scope.go:117] "RemoveContainer" containerID="a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919" Jul 06 15:30:12.859517 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.859497 2561 scope.go:117] "RemoveContainer" containerID="3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4" Jul 06 15:30:12.859774 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:30:12.859755 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4\": container with ID starting with 3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4 not found: ID does not exist" containerID="3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4" Jul 06 15:30:12.859817 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.859784 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4"} err="failed to get container status \"3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4\": rpc error: code = NotFound desc = could not find container \"3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4\": container with ID starting with 3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4 not found: ID does not exist" Jul 06 15:30:12.859817 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.859803 2561 scope.go:117] "RemoveContainer" containerID="4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318" Jul 06 15:30:12.860059 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:30:12.860043 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318\": container with ID starting with 4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318 not found: ID does not exist" containerID="4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318" Jul 06 15:30:12.860106 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.860064 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318"} err="failed to get container status \"4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318\": rpc error: code = NotFound desc = could not find container \"4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318\": container with ID starting with 4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318 not found: ID does not exist" Jul 06 15:30:12.860106 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.860079 2561 scope.go:117] "RemoveContainer" containerID="093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9" Jul 06 15:30:12.860326 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:30:12.860308 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9\": container with ID starting with 093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9 not found: ID does not exist" containerID="093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9" Jul 06 15:30:12.860389 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.860329 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9"} err="failed to get container status \"093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9\": rpc error: code = NotFound desc = could not find container \"093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9\": container with ID starting with 093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9 not found: ID does not exist" Jul 06 15:30:12.860389 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.860343 2561 scope.go:117] "RemoveContainer" containerID="758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22" Jul 06 15:30:12.860601 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:30:12.860577 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22\": container with ID starting with 758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22 not found: ID does not exist" containerID="758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22" Jul 06 15:30:12.860683 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.860608 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22"} err="failed to get container status \"758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22\": rpc error: code = NotFound desc = could not find container \"758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22\": container with ID starting with 758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22 not found: ID does not exist" Jul 06 15:30:12.860683 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.860626 2561 scope.go:117] "RemoveContainer" containerID="f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f" Jul 06 15:30:12.860917 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:30:12.860897 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f\": container with ID starting with f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f not found: ID does not exist" containerID="f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f" Jul 06 15:30:12.861004 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.860925 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f"} err="failed to get container status \"f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f\": rpc error: code = NotFound desc = could not find container \"f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f\": container with ID starting with f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f not found: ID does not exist" Jul 06 15:30:12.861004 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.860945 2561 scope.go:117] "RemoveContainer" containerID="d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a" Jul 06 15:30:12.861164 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:30:12.861149 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a\": container with ID starting with d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a not found: ID does not exist" containerID="d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a" Jul 06 15:30:12.861222 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.861171 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a"} err="failed to get container status \"d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a\": rpc error: code = NotFound desc = could not find container \"d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a\": container with ID starting with d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a not found: ID does not exist" Jul 06 15:30:12.861222 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.861189 2561 scope.go:117] "RemoveContainer" containerID="a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919" Jul 06 15:30:12.861445 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:30:12.861429 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919\": container with ID starting with a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919 not found: ID does not exist" containerID="a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919" Jul 06 15:30:12.861485 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.861450 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919"} err="failed to get container status \"a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919\": rpc error: code = NotFound desc = could not find container \"a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919\": container with ID starting with a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919 not found: ID does not exist" Jul 06 15:30:12.861485 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.861463 2561 scope.go:117] "RemoveContainer" containerID="3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4" Jul 06 15:30:12.861791 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.861773 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4"} err="failed to get container status \"3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4\": rpc error: code = NotFound desc = could not find container \"3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4\": container with ID starting with 3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4 not found: ID does not exist" Jul 06 15:30:12.861791 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.861791 2561 scope.go:117] "RemoveContainer" containerID="4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318" Jul 06 15:30:12.861994 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.861975 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318"} err="failed to get container status \"4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318\": rpc error: code = NotFound desc = could not find container \"4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318\": container with ID starting with 4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318 not found: ID does not exist" Jul 06 15:30:12.861994 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.861994 2561 scope.go:117] "RemoveContainer" containerID="093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9" Jul 06 15:30:12.862207 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.862192 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9"} err="failed to get container status \"093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9\": rpc error: code = NotFound desc = could not find container \"093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9\": container with ID starting with 093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9 not found: ID does not exist" Jul 06 15:30:12.862254 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.862207 2561 scope.go:117] "RemoveContainer" containerID="758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22" Jul 06 15:30:12.862434 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.862416 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22"} err="failed to get container status \"758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22\": rpc error: code = NotFound desc = could not find container \"758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22\": container with ID starting with 758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22 not found: ID does not exist" Jul 06 15:30:12.862485 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.862436 2561 scope.go:117] "RemoveContainer" containerID="f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f" Jul 06 15:30:12.862661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.862645 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f"} err="failed to get container status \"f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f\": rpc error: code = NotFound desc = could not find container \"f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f\": container with ID starting with f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f not found: ID does not exist" Jul 06 15:30:12.862712 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.862662 2561 scope.go:117] "RemoveContainer" containerID="d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a" Jul 06 15:30:12.862835 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.862817 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a"} err="failed to get container status \"d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a\": rpc error: code = NotFound desc = could not find container \"d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a\": container with ID starting with d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a not found: ID does not exist" Jul 06 15:30:12.862887 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.862836 2561 scope.go:117] "RemoveContainer" containerID="a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919" Jul 06 15:30:12.863059 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.863041 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919"} err="failed to get container status \"a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919\": rpc error: code = NotFound desc = could not find container \"a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919\": container with ID starting with a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919 not found: ID does not exist" Jul 06 15:30:12.863101 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.863060 2561 scope.go:117] "RemoveContainer" containerID="3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4" Jul 06 15:30:12.863265 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.863249 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4"} err="failed to get container status \"3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4\": rpc error: code = NotFound desc = could not find container \"3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4\": container with ID starting with 3b9bf9f310b517e50a33c7eea1544895b6f6ceb256298c1ead88828a299e27b4 not found: ID does not exist" Jul 06 15:30:12.863301 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.863266 2561 scope.go:117] "RemoveContainer" containerID="4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318" Jul 06 15:30:12.863486 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.863469 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318"} err="failed to get container status \"4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318\": rpc error: code = NotFound desc = could not find container \"4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318\": container with ID starting with 4dd098d93e57d9d2986b99e472eea6a0a65f8476cbae8dff336454b02de42318 not found: ID does not exist" Jul 06 15:30:12.863560 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.863486 2561 scope.go:117] "RemoveContainer" containerID="093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9" Jul 06 15:30:12.863711 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.863690 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9"} err="failed to get container status \"093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9\": rpc error: code = NotFound desc = could not find container \"093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9\": container with ID starting with 093cf62c2dff6a304ab61af664eaa1008fbee53cbd190f4dd21637ab5c3c3be9 not found: ID does not exist" Jul 06 15:30:12.863761 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.863711 2561 scope.go:117] "RemoveContainer" containerID="758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22" Jul 06 15:30:12.863918 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.863902 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22"} err="failed to get container status \"758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22\": rpc error: code = NotFound desc = could not find container \"758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22\": container with ID starting with 758f5ed8d3883b87c7c5e1e1d4c82ca7fbc277e5656833f4cd20d9390da3cc22 not found: ID does not exist" Jul 06 15:30:12.863961 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.863918 2561 scope.go:117] "RemoveContainer" containerID="f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f" Jul 06 15:30:12.864091 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.864075 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f"} err="failed to get container status \"f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f\": rpc error: code = NotFound desc = could not find container \"f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f\": container with ID starting with f7143bc97164841c4b782a9cf043ff329679ea6551a9968cc81fc2660742354f not found: ID does not exist" Jul 06 15:30:12.864162 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.864092 2561 scope.go:117] "RemoveContainer" containerID="d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a" Jul 06 15:30:12.864301 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.864281 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a"} err="failed to get container status \"d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a\": rpc error: code = NotFound desc = could not find container \"d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a\": container with ID starting with d907b6702d9ca4db89491cc3def6e08da6107d3a9987f51bd856abadae6a4c1a not found: ID does not exist" Jul 06 15:30:12.864354 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.864302 2561 scope.go:117] "RemoveContainer" containerID="a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919" Jul 06 15:30:12.864495 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.864480 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919"} err="failed to get container status \"a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919\": rpc error: code = NotFound desc = could not find container \"a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919\": container with ID starting with a79f79c78bdd738c9ae03a5ca4032a51a9ab14d753c5752d7e64528f95c3c919 not found: ID does not exist" Jul 06 15:30:12.869401 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869383 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:30:12.869659 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869647 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="kube-rbac-proxy-thanos" Jul 06 15:30:12.869707 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869661 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="kube-rbac-proxy-thanos" Jul 06 15:30:12.869707 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869672 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d94cdbae-fb8c-404a-bdf4-6fa955bccab9" containerName="console" Jul 06 15:30:12.869707 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869678 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d94cdbae-fb8c-404a-bdf4-6fa955bccab9" containerName="console" Jul 06 15:30:12.869707 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869688 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="thanos-sidecar" Jul 06 15:30:12.869707 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869693 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="thanos-sidecar" Jul 06 15:30:12.869707 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869701 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="config-reloader" Jul 06 15:30:12.869707 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869706 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="config-reloader" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869712 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="kube-rbac-proxy-web" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869717 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="kube-rbac-proxy-web" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869723 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="kube-rbac-proxy" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869728 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="kube-rbac-proxy" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869734 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="prometheus" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869739 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="prometheus" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869745 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2847c482-6468-4427-9cc2-61ebdb91ed68" containerName="console" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869751 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2847c482-6468-4427-9cc2-61ebdb91ed68" containerName="console" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869762 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="init-config-reloader" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869767 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="init-config-reloader" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869823 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="thanos-sidecar" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869830 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="kube-rbac-proxy-web" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869836 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2847c482-6468-4427-9cc2-61ebdb91ed68" containerName="console" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869842 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="kube-rbac-proxy-thanos" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869849 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d94cdbae-fb8c-404a-bdf4-6fa955bccab9" containerName="console" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869855 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="prometheus" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869860 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="kube-rbac-proxy" Jul 06 15:30:12.869917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.869865 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" containerName="config-reloader" Jul 06 15:30:12.872092 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.872077 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:12.874988 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.874941 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 15:30:12.875082 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.875025 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 15:30:12.875082 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.875054 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 15:30:12.875205 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.875150 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 15:30:12.875205 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.875153 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 15:30:12.875388 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.875319 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 15:30:12.875388 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.875355 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 15:30:12.875514 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.875406 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 15:30:12.875676 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.875658 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 15:30:12.875771 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.875757 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 15:30:12.875870 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.875857 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bkgu8e903q43u\"" Jul 06 15:30:12.876357 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.876292 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 15:30:12.876357 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.876295 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-h8ggc\"" Jul 06 15:30:12.876654 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.876376 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 15:30:12.876763 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.876726 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 15:30:12.880681 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.880659 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 15:30:12.882617 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.882601 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 15:30:12.901477 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:12.901458 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:30:13.038318 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038291 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038432 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038328 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038432 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038347 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038432 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038364 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038432 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038381 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038432 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038419 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038639 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038435 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzxk4\" (UniqueName: \"kubernetes.io/projected/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-kube-api-access-mzxk4\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038639 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038465 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-config\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038639 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038477 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038639 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038492 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038639 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038508 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038639 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038547 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038639 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038585 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038639 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038602 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038639 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038621 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038639 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038637 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038927 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038672 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038927 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038705 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038927 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038738 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.038927 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.038764 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.139745 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.139723 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.139841 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.139752 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.139841 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.139768 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.139841 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.139787 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.139987 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.139914 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.139987 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.139944 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.139987 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.139962 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzxk4\" (UniqueName: \"kubernetes.io/projected/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-kube-api-access-mzxk4\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140126 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140020 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-config\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140126 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140057 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140126 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140086 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140126 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140110 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140308 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140136 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140308 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140162 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140308 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140193 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140308 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140225 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140308 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140230 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140308 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140251 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140308 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140276 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140308 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140305 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140726 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140342 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140726 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140393 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140726 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140513 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.140726 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140558 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.141084 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.140775 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.141750 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.141723 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.142757 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.142618 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.143500 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.143478 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.143754 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.143731 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.143961 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.143935 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-config\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.144433 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.144331 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.144692 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.144674 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.145257 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.145184 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.145257 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.145237 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.145560 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.145518 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.145871 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.145830 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.145871 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.145854 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.146179 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.146146 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.146309 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.146291 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.146958 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.146937 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.150112 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.150091 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzxk4\" (UniqueName: \"kubernetes.io/projected/0f8157a4-8adc-4a02-a4ca-c4b2989a7a30-kube-api-access-mzxk4\") pod \"prometheus-k8s-0\" (UID: \"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.182789 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.182770 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:13.335641 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.335617 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:30:13.337851 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:30:13.337826 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8157a4_8adc_4a02_a4ca_c4b2989a7a30.slice/crio-e9e5656919c837c41d8e556f2487d0f74c917bf42a46a95029a90e84f79b2fb3 WatchSource:0}: Error finding container e9e5656919c837c41d8e556f2487d0f74c917bf42a46a95029a90e84f79b2fb3: Status 404 returned error can't find the container with id e9e5656919c837c41d8e556f2487d0f74c917bf42a46a95029a90e84f79b2fb3 Jul 06 15:30:13.814472 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.814439 2561 generic.go:356] "Generic (PLEG): container finished" podID="0f8157a4-8adc-4a02-a4ca-c4b2989a7a30" containerID="96d11f961af469679baff5f6bbd4a52ca3f35ce4fb3f347d02f1710fdcf6c047" exitCode=0 Jul 06 15:30:13.814872 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.814476 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30","Type":"ContainerDied","Data":"96d11f961af469679baff5f6bbd4a52ca3f35ce4fb3f347d02f1710fdcf6c047"} Jul 06 15:30:13.814872 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:13.814561 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30","Type":"ContainerStarted","Data":"e9e5656919c837c41d8e556f2487d0f74c917bf42a46a95029a90e84f79b2fb3"} Jul 06 15:30:14.799128 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:14.799099 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6f83a99-4b1c-424d-a6bd-f3f10070b14f" path="/var/lib/kubelet/pods/e6f83a99-4b1c-424d-a6bd-f3f10070b14f/volumes" Jul 06 15:30:14.821827 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:14.821800 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30","Type":"ContainerStarted","Data":"4007a885746b92391f615eae3eb8879562863a032a1f35b60963f2fae24a114d"} Jul 06 15:30:14.822152 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:14.821855 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30","Type":"ContainerStarted","Data":"cee216fcd5fc223447f275a234aea8be151b5e742424f9b52ffdca840c07506d"} Jul 06 15:30:14.822152 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:14.821881 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30","Type":"ContainerStarted","Data":"0d807c63203a9267cdec31dc76e1b64830f4245df27891461b8ed3e54afc2449"} Jul 06 15:30:14.822152 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:14.821906 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30","Type":"ContainerStarted","Data":"8fb9982725d992e8ac9a6204006eb975173245bba7172b3d5b1d6df3bb4b3031"} Jul 06 15:30:14.822152 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:14.821929 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30","Type":"ContainerStarted","Data":"7b3563b63ce96304202abcd919f04129ba4d44cd9b3fc8331a9cb082dde60dbc"} Jul 06 15:30:14.822152 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:14.821953 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f8157a4-8adc-4a02-a4ca-c4b2989a7a30","Type":"ContainerStarted","Data":"f3166ba8365ec3a550afeb8e4b4419d3a4b502d302e12b013a5f5f095268c217"} Jul 06 15:30:14.867720 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:14.867672 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.8489072159999997 podStartE2EDuration="2.867655698s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.748482ms" isStatefulPod=false podCreationTimestamp="2026-07-06 15:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:30:14.864612133 +0000 UTC m=+236.497164116" watchObservedRunningTime="2026-07-06 15:30:14.867655698 +0000 UTC m=+236.500207681" Jul 06 15:30:15.016309 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:15.016281 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:15.171890 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:15.171813 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:18.183568 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:18.183517 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:30:18.302817 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:18.302782 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:18.387905 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:18.387870 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:23.611261 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:23.611165 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:23.774409 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:23.774377 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:28.973497 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:28.973461 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:29.103082 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:29.103045 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:34.316837 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:34.316800 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:34.415910 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:34.415876 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:30:34.512211 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:30:34.512178 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:13.183252 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:13.183221 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:31:13.199056 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:13.199026 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:31:13.928681 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:13.928647 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:14.064361 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:14.064336 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:31:14.104971 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:14.104937 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:18.702295 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:18.702267 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 15:31:18.707272 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:18.707251 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 15:31:18.713826 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:18.713808 2561 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 15:31:23.464978 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:23.464940 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:23.567446 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:23.567420 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:27.804236 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:27.804200 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:27.905873 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:27.905834 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:50.068801 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:50.068766 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:50.167740 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:50.167709 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:52.891239 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:52.891202 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:54.527317 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:54.527288 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:55.492383 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:55.492338 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:58.141737 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:58.141700 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:58.273842 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:58.273803 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:31:59.886070 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:31:59.886033 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:00.367188 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:00.367152 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:00.521608 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:00.521570 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:02.024022 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:02.023994 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:02.634673 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:02.634638 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:05.142354 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:05.142317 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:05.259088 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:05.259058 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:11.454594 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:11.454501 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:13.073136 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:13.073099 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:13.252430 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:13.252397 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:16.531347 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:16.531310 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:16.778833 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:16.778796 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:29.455147 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:29.455109 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:29.547275 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:29.547248 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:31.665104 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:31.665046 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:47.194936 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:47.194885 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:47.299471 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:47.299423 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:47.987300 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:47.987258 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:48.285303 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.285210 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:48.411465 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.411435 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:48.573050 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.572985 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c"] Jul 06 15:32:48.576240 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.576224 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:48.583259 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:32:48.583235 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-129-225.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-225.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" type="*v1.Secret" Jul 06 15:32:48.583481 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:32:48.583462 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-225.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-225.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 06 15:32:48.583714 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:32:48.583697 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-dockercfg-d8sts\" is forbidden: User \"system:node:ip-10-0-129-225.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-225.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-d8sts\"" type="*v1.Secret" Jul 06 15:32:48.588178 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.588164 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 15:32:48.594280 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.594266 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 15:32:48.635100 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.635070 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c"] Jul 06 15:32:48.685493 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.685463 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36396d1c-372d-4146-94a7-59e1c7003e40-manifests\") pod \"opendatahub-operator-controller-manager-5bdc85857f-f2l8c\" (UID: \"36396d1c-372d-4146-94a7-59e1c7003e40\") " pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:48.685635 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.685516 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36396d1c-372d-4146-94a7-59e1c7003e40-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bdc85857f-f2l8c\" (UID: \"36396d1c-372d-4146-94a7-59e1c7003e40\") " pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:48.685635 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.685568 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36396d1c-372d-4146-94a7-59e1c7003e40-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bdc85857f-f2l8c\" (UID: \"36396d1c-372d-4146-94a7-59e1c7003e40\") " pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:48.685723 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.685635 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36396d1c-372d-4146-94a7-59e1c7003e40-tmp\") pod \"opendatahub-operator-controller-manager-5bdc85857f-f2l8c\" (UID: \"36396d1c-372d-4146-94a7-59e1c7003e40\") " pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:48.685723 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.685682 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rlms\" (UniqueName: \"kubernetes.io/projected/36396d1c-372d-4146-94a7-59e1c7003e40-kube-api-access-9rlms\") pod \"opendatahub-operator-controller-manager-5bdc85857f-f2l8c\" (UID: \"36396d1c-372d-4146-94a7-59e1c7003e40\") " pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:48.786824 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.786801 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36396d1c-372d-4146-94a7-59e1c7003e40-manifests\") pod \"opendatahub-operator-controller-manager-5bdc85857f-f2l8c\" (UID: \"36396d1c-372d-4146-94a7-59e1c7003e40\") " pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:48.786946 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.786842 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36396d1c-372d-4146-94a7-59e1c7003e40-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bdc85857f-f2l8c\" (UID: \"36396d1c-372d-4146-94a7-59e1c7003e40\") " pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:48.786946 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.786867 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36396d1c-372d-4146-94a7-59e1c7003e40-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bdc85857f-f2l8c\" (UID: \"36396d1c-372d-4146-94a7-59e1c7003e40\") " pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:48.786946 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.786930 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36396d1c-372d-4146-94a7-59e1c7003e40-tmp\") pod \"opendatahub-operator-controller-manager-5bdc85857f-f2l8c\" (UID: \"36396d1c-372d-4146-94a7-59e1c7003e40\") " pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:48.787063 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.786959 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rlms\" (UniqueName: \"kubernetes.io/projected/36396d1c-372d-4146-94a7-59e1c7003e40-kube-api-access-9rlms\") pod \"opendatahub-operator-controller-manager-5bdc85857f-f2l8c\" (UID: \"36396d1c-372d-4146-94a7-59e1c7003e40\") " pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:48.787264 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.787246 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/36396d1c-372d-4146-94a7-59e1c7003e40-manifests\") pod \"opendatahub-operator-controller-manager-5bdc85857f-f2l8c\" (UID: \"36396d1c-372d-4146-94a7-59e1c7003e40\") " pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:48.789365 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.789335 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36396d1c-372d-4146-94a7-59e1c7003e40-tmp\") pod \"opendatahub-operator-controller-manager-5bdc85857f-f2l8c\" (UID: \"36396d1c-372d-4146-94a7-59e1c7003e40\") " pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:48.789490 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.789476 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36396d1c-372d-4146-94a7-59e1c7003e40-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bdc85857f-f2l8c\" (UID: \"36396d1c-372d-4146-94a7-59e1c7003e40\") " pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:48.789662 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.789641 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/36396d1c-372d-4146-94a7-59e1c7003e40-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bdc85857f-f2l8c\" (UID: \"36396d1c-372d-4146-94a7-59e1c7003e40\") " pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:48.876943 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.876873 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:48.980544 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:48.980494 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:49.089244 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:49.089212 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:49.207898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:49.207823 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:49.344918 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:49.344870 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:49.448354 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:49.448329 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:49.558814 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:49.558789 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 15:32:49.565604 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:49.565576 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 15:32:49.572961 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:49.572942 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rlms\" (UniqueName: \"kubernetes.io/projected/36396d1c-372d-4146-94a7-59e1c7003e40-kube-api-access-9rlms\") pod \"opendatahub-operator-controller-manager-5bdc85857f-f2l8c\" (UID: \"36396d1c-372d-4146-94a7-59e1c7003e40\") " pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:50.128401 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:50.128370 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:50.156817 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:50.156789 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-d8sts\"" Jul 06 15:32:50.166468 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:50.166451 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:50.232242 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:50.231287 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:32:50.321727 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:50.321702 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c"] Jul 06 15:32:50.323302 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:32:50.323274 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36396d1c_372d_4146_94a7_59e1c7003e40.slice/crio-7e580976e9a4c2c1e089e7497ff6ceafed792e8c716385c33fef0819ccf9732f WatchSource:0}: Error finding container 7e580976e9a4c2c1e089e7497ff6ceafed792e8c716385c33fef0819ccf9732f: Status 404 returned error can't find the container with id 7e580976e9a4c2c1e089e7497ff6ceafed792e8c716385c33fef0819ccf9732f Jul 06 15:32:50.325337 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:50.325320 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 15:32:50.420425 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:50.420336 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" event={"ID":"36396d1c-372d-4146-94a7-59e1c7003e40","Type":"ContainerStarted","Data":"7e580976e9a4c2c1e089e7497ff6ceafed792e8c716385c33fef0819ccf9732f"} Jul 06 15:32:53.437231 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:53.437146 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" event={"ID":"36396d1c-372d-4146-94a7-59e1c7003e40","Type":"ContainerStarted","Data":"b2488f695eef9a3ebc9d196c2bc772063950fa57aaacb47dc760722c275bd41b"} Jul 06 15:32:54.443853 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:54.443818 2561 generic.go:356] "Generic (PLEG): container finished" podID="36396d1c-372d-4146-94a7-59e1c7003e40" containerID="b2488f695eef9a3ebc9d196c2bc772063950fa57aaacb47dc760722c275bd41b" exitCode=0 Jul 06 15:32:54.444266 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:54.443900 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" event={"ID":"36396d1c-372d-4146-94a7-59e1c7003e40","Type":"ContainerDied","Data":"b2488f695eef9a3ebc9d196c2bc772063950fa57aaacb47dc760722c275bd41b"} Jul 06 15:32:55.450754 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:55.450721 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" event={"ID":"36396d1c-372d-4146-94a7-59e1c7003e40","Type":"ContainerStarted","Data":"41751fd48c4085781b17c1b81678ebd948dfb2c2d3c53190d67786e924240f9a"} Jul 06 15:32:55.451207 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:55.450866 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:32:55.485381 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:32:55.485340 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" podStartSLOduration=3.185099085 podStartE2EDuration="7.485325719s" totalImagesPullingTime="3.208611378s" totalInitContainerRuntime="1.091615256s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:32:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:32:55.484086769 +0000 UTC m=+397.116638765" watchObservedRunningTime="2026-07-06 15:32:55.485325719 +0000 UTC m=+397.117877702" Jul 06 15:33:00.184789 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:00.184759 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:00.305197 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:00.305155 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:06.457592 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:06.457564 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5bdc85857f-f2l8c" Jul 06 15:33:08.549889 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:08.549853 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:08.681748 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:08.681716 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:09.453341 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:09.453309 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:09.533262 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:09.533227 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:09.634173 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:09.634142 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:09.854957 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:09.854914 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:10.131431 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:10.131335 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:10.512354 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:10.512315 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:12.122162 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:12.122126 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:12.385058 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:12.384955 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:13.416492 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:13.416444 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:13.515630 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:13.515599 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:14.511586 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:14.511554 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:19.592440 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.592411 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-df4586b48-bldnw"] Jul 06 15:33:19.597200 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.597180 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:19.602061 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.602034 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 15:33:19.602061 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.602052 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tgbnv\"" Jul 06 15:33:19.602260 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.602106 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:33:19.602260 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.602122 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 15:33:19.602260 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.602041 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 15:33:19.602260 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.602165 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 15:33:19.609605 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.609577 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-df4586b48-bldnw"] Jul 06 15:33:19.638032 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.638011 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a4caab86-7302-4ef7-8241-7dea80d7ef13-manager-config\") pod \"lws-controller-manager-df4586b48-bldnw\" (UID: \"a4caab86-7302-4ef7-8241-7dea80d7ef13\") " pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:19.638160 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.638045 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a4caab86-7302-4ef7-8241-7dea80d7ef13-metrics-cert\") pod \"lws-controller-manager-df4586b48-bldnw\" (UID: \"a4caab86-7302-4ef7-8241-7dea80d7ef13\") " pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:19.638160 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.638072 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmfxg\" (UniqueName: \"kubernetes.io/projected/a4caab86-7302-4ef7-8241-7dea80d7ef13-kube-api-access-gmfxg\") pod \"lws-controller-manager-df4586b48-bldnw\" (UID: \"a4caab86-7302-4ef7-8241-7dea80d7ef13\") " pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:19.638278 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.638191 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4caab86-7302-4ef7-8241-7dea80d7ef13-cert\") pod \"lws-controller-manager-df4586b48-bldnw\" (UID: \"a4caab86-7302-4ef7-8241-7dea80d7ef13\") " pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:19.739038 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.739012 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4caab86-7302-4ef7-8241-7dea80d7ef13-cert\") pod \"lws-controller-manager-df4586b48-bldnw\" (UID: \"a4caab86-7302-4ef7-8241-7dea80d7ef13\") " pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:19.739173 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.739051 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a4caab86-7302-4ef7-8241-7dea80d7ef13-manager-config\") pod \"lws-controller-manager-df4586b48-bldnw\" (UID: \"a4caab86-7302-4ef7-8241-7dea80d7ef13\") " pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:19.739173 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.739079 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a4caab86-7302-4ef7-8241-7dea80d7ef13-metrics-cert\") pod \"lws-controller-manager-df4586b48-bldnw\" (UID: \"a4caab86-7302-4ef7-8241-7dea80d7ef13\") " pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:19.739173 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.739109 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmfxg\" (UniqueName: \"kubernetes.io/projected/a4caab86-7302-4ef7-8241-7dea80d7ef13-kube-api-access-gmfxg\") pod \"lws-controller-manager-df4586b48-bldnw\" (UID: \"a4caab86-7302-4ef7-8241-7dea80d7ef13\") " pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:19.739671 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.739652 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a4caab86-7302-4ef7-8241-7dea80d7ef13-manager-config\") pod \"lws-controller-manager-df4586b48-bldnw\" (UID: \"a4caab86-7302-4ef7-8241-7dea80d7ef13\") " pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:19.741564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.741544 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a4caab86-7302-4ef7-8241-7dea80d7ef13-metrics-cert\") pod \"lws-controller-manager-df4586b48-bldnw\" (UID: \"a4caab86-7302-4ef7-8241-7dea80d7ef13\") " pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:19.741679 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.741663 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a4caab86-7302-4ef7-8241-7dea80d7ef13-cert\") pod \"lws-controller-manager-df4586b48-bldnw\" (UID: \"a4caab86-7302-4ef7-8241-7dea80d7ef13\") " pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:19.757333 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.757314 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmfxg\" (UniqueName: \"kubernetes.io/projected/a4caab86-7302-4ef7-8241-7dea80d7ef13-kube-api-access-gmfxg\") pod \"lws-controller-manager-df4586b48-bldnw\" (UID: \"a4caab86-7302-4ef7-8241-7dea80d7ef13\") " pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:19.906951 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:19.906885 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:20.052632 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.052591 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:20.068219 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.068193 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-df4586b48-bldnw"] Jul 06 15:33:20.085456 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:33:20.085432 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4caab86_7302_4ef7_8241_7dea80d7ef13.slice/crio-68493aa3f7f99bdaf8bba385e67c0d7f53bc94e916992b95d2d01acd38edef67 WatchSource:0}: Error finding container 68493aa3f7f99bdaf8bba385e67c0d7f53bc94e916992b95d2d01acd38edef67: Status 404 returned error can't find the container with id 68493aa3f7f99bdaf8bba385e67c0d7f53bc94e916992b95d2d01acd38edef67 Jul 06 15:33:20.171369 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.171300 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:20.228888 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.228861 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nktks"] Jul 06 15:33:20.233234 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.233216 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nktks" Jul 06 15:33:20.242033 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.241939 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 15:33:20.242203 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.242184 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-lzjsm\"" Jul 06 15:33:20.242818 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.242797 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b"] Jul 06 15:33:20.246342 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.246326 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" Jul 06 15:33:20.256693 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.256676 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-m2swp\"" Jul 06 15:33:20.256793 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.256720 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 15:33:20.282336 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.282311 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b"] Jul 06 15:33:20.284437 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.284414 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nktks"] Jul 06 15:33:20.289484 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.289460 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:20.345711 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.345679 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfjjw\" (UniqueName: \"kubernetes.io/projected/0bb58fc6-57b0-4511-bb64-ca0a43d05ee0-kube-api-access-cfjjw\") pod \"kserve-controller-manager-d6f4b65b8-nktks\" (UID: \"0bb58fc6-57b0-4511-bb64-ca0a43d05ee0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nktks" Jul 06 15:33:20.345909 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.345740 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bb58fc6-57b0-4511-bb64-ca0a43d05ee0-cert\") pod \"kserve-controller-manager-d6f4b65b8-nktks\" (UID: \"0bb58fc6-57b0-4511-bb64-ca0a43d05ee0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nktks" Jul 06 15:33:20.345909 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.345768 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf9s4\" (UniqueName: \"kubernetes.io/projected/46d03c46-7d6d-4084-8b9a-211fec671746-kube-api-access-hf9s4\") pod \"llmisvc-controller-manager-777c6bd7d5-vhd2b\" (UID: \"46d03c46-7d6d-4084-8b9a-211fec671746\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" Jul 06 15:33:20.345909 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.345858 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46d03c46-7d6d-4084-8b9a-211fec671746-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vhd2b\" (UID: \"46d03c46-7d6d-4084-8b9a-211fec671746\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" Jul 06 15:33:20.446589 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.446486 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46d03c46-7d6d-4084-8b9a-211fec671746-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vhd2b\" (UID: \"46d03c46-7d6d-4084-8b9a-211fec671746\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" Jul 06 15:33:20.446589 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.446558 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfjjw\" (UniqueName: \"kubernetes.io/projected/0bb58fc6-57b0-4511-bb64-ca0a43d05ee0-kube-api-access-cfjjw\") pod \"kserve-controller-manager-d6f4b65b8-nktks\" (UID: \"0bb58fc6-57b0-4511-bb64-ca0a43d05ee0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nktks" Jul 06 15:33:20.446785 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.446608 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bb58fc6-57b0-4511-bb64-ca0a43d05ee0-cert\") pod \"kserve-controller-manager-d6f4b65b8-nktks\" (UID: \"0bb58fc6-57b0-4511-bb64-ca0a43d05ee0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nktks" Jul 06 15:33:20.446785 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.446632 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf9s4\" (UniqueName: \"kubernetes.io/projected/46d03c46-7d6d-4084-8b9a-211fec671746-kube-api-access-hf9s4\") pod \"llmisvc-controller-manager-777c6bd7d5-vhd2b\" (UID: \"46d03c46-7d6d-4084-8b9a-211fec671746\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" Jul 06 15:33:20.446785 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:33:20.446659 2561 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 15:33:20.446785 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:33:20.446709 2561 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 06 15:33:20.446785 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:33:20.446725 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d03c46-7d6d-4084-8b9a-211fec671746-cert podName:46d03c46-7d6d-4084-8b9a-211fec671746 nodeName:}" failed. No retries permitted until 2026-07-06 15:33:20.9467054 +0000 UTC m=+422.579257373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/46d03c46-7d6d-4084-8b9a-211fec671746-cert") pod "llmisvc-controller-manager-777c6bd7d5-vhd2b" (UID: "46d03c46-7d6d-4084-8b9a-211fec671746") : secret "llmisvc-webhook-server-cert" not found Jul 06 15:33:20.446785 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:33:20.446761 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb58fc6-57b0-4511-bb64-ca0a43d05ee0-cert podName:0bb58fc6-57b0-4511-bb64-ca0a43d05ee0 nodeName:}" failed. No retries permitted until 2026-07-06 15:33:20.946745514 +0000 UTC m=+422.579297482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0bb58fc6-57b0-4511-bb64-ca0a43d05ee0-cert") pod "kserve-controller-manager-d6f4b65b8-nktks" (UID: "0bb58fc6-57b0-4511-bb64-ca0a43d05ee0") : secret "kserve-webhook-server-cert" not found Jul 06 15:33:20.465971 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.465947 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfjjw\" (UniqueName: \"kubernetes.io/projected/0bb58fc6-57b0-4511-bb64-ca0a43d05ee0-kube-api-access-cfjjw\") pod \"kserve-controller-manager-d6f4b65b8-nktks\" (UID: \"0bb58fc6-57b0-4511-bb64-ca0a43d05ee0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nktks" Jul 06 15:33:20.466156 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.466133 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf9s4\" (UniqueName: \"kubernetes.io/projected/46d03c46-7d6d-4084-8b9a-211fec671746-kube-api-access-hf9s4\") pod \"llmisvc-controller-manager-777c6bd7d5-vhd2b\" (UID: \"46d03c46-7d6d-4084-8b9a-211fec671746\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" Jul 06 15:33:20.513786 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.513719 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:20.555580 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.555521 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" event={"ID":"a4caab86-7302-4ef7-8241-7dea80d7ef13","Type":"ContainerStarted","Data":"68493aa3f7f99bdaf8bba385e67c0d7f53bc94e916992b95d2d01acd38edef67"} Jul 06 15:33:20.617312 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.617276 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:20.951074 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.951034 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46d03c46-7d6d-4084-8b9a-211fec671746-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vhd2b\" (UID: \"46d03c46-7d6d-4084-8b9a-211fec671746\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" Jul 06 15:33:20.951240 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.951120 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bb58fc6-57b0-4511-bb64-ca0a43d05ee0-cert\") pod \"kserve-controller-manager-d6f4b65b8-nktks\" (UID: \"0bb58fc6-57b0-4511-bb64-ca0a43d05ee0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nktks" Jul 06 15:33:20.951240 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:33:20.951180 2561 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 06 15:33:20.951329 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:33:20.951241 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d03c46-7d6d-4084-8b9a-211fec671746-cert podName:46d03c46-7d6d-4084-8b9a-211fec671746 nodeName:}" failed. No retries permitted until 2026-07-06 15:33:21.951226876 +0000 UTC m=+423.583778838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/46d03c46-7d6d-4084-8b9a-211fec671746-cert") pod "llmisvc-controller-manager-777c6bd7d5-vhd2b" (UID: "46d03c46-7d6d-4084-8b9a-211fec671746") : secret "llmisvc-webhook-server-cert" not found Jul 06 15:33:20.953792 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:20.953764 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bb58fc6-57b0-4511-bb64-ca0a43d05ee0-cert\") pod \"kserve-controller-manager-d6f4b65b8-nktks\" (UID: \"0bb58fc6-57b0-4511-bb64-ca0a43d05ee0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nktks" Jul 06 15:33:21.146384 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.146352 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nktks" Jul 06 15:33:21.178850 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.178782 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:21.318188 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.318161 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nktks"] Jul 06 15:33:21.320289 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:33:21.320259 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb58fc6_57b0_4511_bb64_ca0a43d05ee0.slice/crio-c72d1650fb45138d56a5494079cedbfd39e787c3c1b33b08cbb85ce74e6dffe1 WatchSource:0}: Error finding container c72d1650fb45138d56a5494079cedbfd39e787c3c1b33b08cbb85ce74e6dffe1: Status 404 returned error can't find the container with id c72d1650fb45138d56a5494079cedbfd39e787c3c1b33b08cbb85ce74e6dffe1 Jul 06 15:33:21.471918 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.471881 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:21.569268 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.569226 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nktks" event={"ID":"0bb58fc6-57b0-4511-bb64-ca0a43d05ee0","Type":"ContainerStarted","Data":"c72d1650fb45138d56a5494079cedbfd39e787c3c1b33b08cbb85ce74e6dffe1"} Jul 06 15:33:21.576010 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.575981 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:21.661011 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.660967 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:21.808842 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.808758 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22"] Jul 06 15:33:21.812169 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.812149 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.822043 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.822020 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:21.822497 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.822476 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 15:33:21.822791 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.822763 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 15:33:21.824310 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.824293 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-cwrrh\"" Jul 06 15:33:21.824421 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.824292 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 15:33:21.836638 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.836620 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22"] Jul 06 15:33:21.860385 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.860361 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874c3000-ae24-4343-9385-7e7e79ccfa33-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.860543 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.860398 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874c3000-ae24-4343-9385-7e7e79ccfa33-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.860543 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.860449 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874c3000-ae24-4343-9385-7e7e79ccfa33-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.860543 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.860474 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874c3000-ae24-4343-9385-7e7e79ccfa33-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.860700 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.860523 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874c3000-ae24-4343-9385-7e7e79ccfa33-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.860700 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.860591 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874c3000-ae24-4343-9385-7e7e79ccfa33-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.860700 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.860612 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhnqs\" (UniqueName: \"kubernetes.io/projected/874c3000-ae24-4343-9385-7e7e79ccfa33-kube-api-access-vhnqs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.860700 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.860643 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874c3000-ae24-4343-9385-7e7e79ccfa33-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.860700 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.860659 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874c3000-ae24-4343-9385-7e7e79ccfa33-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.961723 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.961697 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874c3000-ae24-4343-9385-7e7e79ccfa33-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.961858 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.961734 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874c3000-ae24-4343-9385-7e7e79ccfa33-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.961858 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.961765 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874c3000-ae24-4343-9385-7e7e79ccfa33-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.961858 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.961791 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874c3000-ae24-4343-9385-7e7e79ccfa33-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.961858 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.961845 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874c3000-ae24-4343-9385-7e7e79ccfa33-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.962069 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.961881 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46d03c46-7d6d-4084-8b9a-211fec671746-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vhd2b\" (UID: \"46d03c46-7d6d-4084-8b9a-211fec671746\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" Jul 06 15:33:21.962069 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.961909 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874c3000-ae24-4343-9385-7e7e79ccfa33-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.962069 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.961934 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhnqs\" (UniqueName: \"kubernetes.io/projected/874c3000-ae24-4343-9385-7e7e79ccfa33-kube-api-access-vhnqs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.962069 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.961971 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874c3000-ae24-4343-9385-7e7e79ccfa33-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.962069 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.961997 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874c3000-ae24-4343-9385-7e7e79ccfa33-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.962324 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.962189 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/874c3000-ae24-4343-9385-7e7e79ccfa33-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.962502 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.962413 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/874c3000-ae24-4343-9385-7e7e79ccfa33-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.962646 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.962589 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/874c3000-ae24-4343-9385-7e7e79ccfa33-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.962739 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.962638 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/874c3000-ae24-4343-9385-7e7e79ccfa33-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.963190 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.963167 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/874c3000-ae24-4343-9385-7e7e79ccfa33-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.964337 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.964308 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/874c3000-ae24-4343-9385-7e7e79ccfa33-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.964513 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.964498 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/874c3000-ae24-4343-9385-7e7e79ccfa33-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.964851 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.964831 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46d03c46-7d6d-4084-8b9a-211fec671746-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vhd2b\" (UID: \"46d03c46-7d6d-4084-8b9a-211fec671746\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" Jul 06 15:33:21.983676 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.983647 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/874c3000-ae24-4343-9385-7e7e79ccfa33-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:21.984630 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:21.984609 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhnqs\" (UniqueName: \"kubernetes.io/projected/874c3000-ae24-4343-9385-7e7e79ccfa33-kube-api-access-vhnqs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4gpw22\" (UID: \"874c3000-ae24-4343-9385-7e7e79ccfa33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:22.054850 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:22.054820 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" Jul 06 15:33:22.074118 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:22.074083 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:22.125583 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:22.125555 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:22.170595 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:22.170402 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:22.234331 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:22.234296 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b"] Jul 06 15:33:22.236970 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:33:22.236943 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod46d03c46_7d6d_4084_8b9a_211fec671746.slice/crio-3878cd5c2c3d16fafd07a7a872a9e2b3c6c1f3b84c914be60f359311417dceb2 WatchSource:0}: Error finding container 3878cd5c2c3d16fafd07a7a872a9e2b3c6c1f3b84c914be60f359311417dceb2: Status 404 returned error can't find the container with id 3878cd5c2c3d16fafd07a7a872a9e2b3c6c1f3b84c914be60f359311417dceb2 Jul 06 15:33:22.306182 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:22.306153 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:22.306939 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:22.306911 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22"] Jul 06 15:33:22.388483 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:22.388408 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:22.575763 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:22.575726 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" event={"ID":"46d03c46-7d6d-4084-8b9a-211fec671746","Type":"ContainerStarted","Data":"3878cd5c2c3d16fafd07a7a872a9e2b3c6c1f3b84c914be60f359311417dceb2"} Jul 06 15:33:23.361736 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:33:23.361705 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874c3000_ae24_4343_9385_7e7e79ccfa33.slice/crio-b66e6cb647c723bc37021899ff92e25d42100aa2ddc41f2c77ddcc023a82cb07 WatchSource:0}: Error finding container b66e6cb647c723bc37021899ff92e25d42100aa2ddc41f2c77ddcc023a82cb07: Status 404 returned error can't find the container with id b66e6cb647c723bc37021899ff92e25d42100aa2ddc41f2c77ddcc023a82cb07 Jul 06 15:33:23.583518 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:23.583473 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" event={"ID":"a4caab86-7302-4ef7-8241-7dea80d7ef13","Type":"ContainerStarted","Data":"cbee1eabf292bc62eca419babda2b953f6ee4bdf833aff847a2b051c4014a5b0"} Jul 06 15:33:23.583707 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:23.583638 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:23.585083 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:23.585053 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" event={"ID":"874c3000-ae24-4343-9385-7e7e79ccfa33","Type":"ContainerStarted","Data":"b66e6cb647c723bc37021899ff92e25d42100aa2ddc41f2c77ddcc023a82cb07"} Jul 06 15:33:23.607008 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:23.606949 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" podStartSLOduration=1.272401832 podStartE2EDuration="4.606926976s" totalImagesPullingTime="3.334525144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:33:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:33:23.604467439 +0000 UTC m=+425.237019426" watchObservedRunningTime="2026-07-06 15:33:23.606926976 +0000 UTC m=+425.239478961" Jul 06 15:33:25.599289 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:25.599247 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nktks" event={"ID":"0bb58fc6-57b0-4511-bb64-ca0a43d05ee0","Type":"ContainerStarted","Data":"81f4009c928384491d5ca6f1b685930ae74ea67ef9a96d99f5f08fb3768597d9"} Jul 06 15:33:25.599725 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:25.599346 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nktks" Jul 06 15:33:25.622073 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:25.622020 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nktks" podStartSLOduration=2.449415147 podStartE2EDuration="5.622004752s" totalImagesPullingTime="3.172589605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:33:25.620677662 +0000 UTC m=+427.253229644" watchObservedRunningTime="2026-07-06 15:33:25.622004752 +0000 UTC m=+427.254556736" Jul 06 15:33:27.973698 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:27.973662 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:33:27.973968 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:27.973737 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:33:27.973968 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:27.973768 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:33:28.616809 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:28.616773 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" event={"ID":"46d03c46-7d6d-4084-8b9a-211fec671746","Type":"ContainerStarted","Data":"6a6df0629fbff2b5d924312bf6dad4258b504ae6cdb0e4a923147166adaf0281"} Jul 06 15:33:28.616994 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:28.616874 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" Jul 06 15:33:28.618228 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:28.618209 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" event={"ID":"874c3000-ae24-4343-9385-7e7e79ccfa33","Type":"ContainerStarted","Data":"c56be4928b1e337b0a6b919a368e09936fd811d0b96656e0c103a6f5927ec296"} Jul 06 15:33:28.637014 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:28.636970 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" podStartSLOduration=2.900058475 podStartE2EDuration="8.636957697s" totalImagesPullingTime="5.736899222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:33:28.635464329 +0000 UTC m=+430.268016313" watchObservedRunningTime="2026-07-06 15:33:28.636957697 +0000 UTC m=+430.269509732" Jul 06 15:33:28.662704 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:28.662634 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" podStartSLOduration=3.053637694 podStartE2EDuration="7.662621958s" totalImagesPullingTime="4.608984264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:33:28.661646465 +0000 UTC m=+430.294198451" watchObservedRunningTime="2026-07-06 15:33:28.662621958 +0000 UTC m=+430.295173940" Jul 06 15:33:29.126243 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:29.126210 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:29.130970 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:29.130946 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:29.622753 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:29.622713 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:29.623492 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:29.623472 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4gpw22" Jul 06 15:33:34.594373 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:34.594343 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-df4586b48-bldnw" Jul 06 15:33:46.134916 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:46.134826 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:33:56.609987 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:56.609958 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nktks" Jul 06 15:33:59.624633 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:33:59.624516 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" Jul 06 15:34:03.327743 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:03.327715 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:10.105928 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.105885 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:10.202096 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.202065 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:10.258928 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.258897 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-686ff6c96c-fkk4w"] Jul 06 15:34:10.269450 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.269429 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.274066 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.274038 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 15:34:10.274199 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.274082 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rmsbt\"" Jul 06 15:34:10.274303 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.274286 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 15:34:10.274471 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.274448 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 15:34:10.274471 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.274444 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 15:34:10.274716 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.274351 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 15:34:10.275003 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.274987 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-686ff6c96c-fkk4w"] Jul 06 15:34:10.279106 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.279084 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 15:34:10.316902 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.316875 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:10.349745 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.349717 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8zmqt"] Jul 06 15:34:10.353604 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.353582 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8zmqt" Jul 06 15:34:10.356761 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.356709 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-f2wbf\"" Jul 06 15:34:10.356761 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.356734 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 15:34:10.356923 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.356900 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 15:34:10.375367 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.375345 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8zmqt"] Jul 06 15:34:10.407560 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.407491 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:10.465175 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.465148 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9d48bff-9a51-48a7-b5f0-30f36e377202-console-oauth-config\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.465302 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.465185 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d48bff-9a51-48a7-b5f0-30f36e377202-service-ca\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.465302 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.465223 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfjvg\" (UniqueName: \"kubernetes.io/projected/a9d48bff-9a51-48a7-b5f0-30f36e377202-kube-api-access-dfjvg\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.465401 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.465318 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9d48bff-9a51-48a7-b5f0-30f36e377202-console-serving-cert\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.465401 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.465344 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9d48bff-9a51-48a7-b5f0-30f36e377202-console-config\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.465401 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.465395 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d48bff-9a51-48a7-b5f0-30f36e377202-trusted-ca-bundle\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.465520 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.465481 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwj6r\" (UniqueName: \"kubernetes.io/projected/b9698f39-22cb-4ab7-b5ac-7b88b7597d49-kube-api-access-bwj6r\") pod \"kuadrant-operator-catalog-8zmqt\" (UID: \"b9698f39-22cb-4ab7-b5ac-7b88b7597d49\") " pod="kuadrant-system/kuadrant-operator-catalog-8zmqt" Jul 06 15:34:10.465590 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.465520 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9d48bff-9a51-48a7-b5f0-30f36e377202-oauth-serving-cert\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.495288 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.495266 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:10.566120 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.566091 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9d48bff-9a51-48a7-b5f0-30f36e377202-console-oauth-config\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.566236 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.566131 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d48bff-9a51-48a7-b5f0-30f36e377202-service-ca\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.566298 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.566241 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfjvg\" (UniqueName: \"kubernetes.io/projected/a9d48bff-9a51-48a7-b5f0-30f36e377202-kube-api-access-dfjvg\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.566298 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.566288 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9d48bff-9a51-48a7-b5f0-30f36e377202-console-serving-cert\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.566421 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.566318 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9d48bff-9a51-48a7-b5f0-30f36e377202-console-config\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.566421 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.566354 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d48bff-9a51-48a7-b5f0-30f36e377202-trusted-ca-bundle\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.566421 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.566401 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwj6r\" (UniqueName: \"kubernetes.io/projected/b9698f39-22cb-4ab7-b5ac-7b88b7597d49-kube-api-access-bwj6r\") pod \"kuadrant-operator-catalog-8zmqt\" (UID: \"b9698f39-22cb-4ab7-b5ac-7b88b7597d49\") " pod="kuadrant-system/kuadrant-operator-catalog-8zmqt" Jul 06 15:34:10.566649 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.566438 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9d48bff-9a51-48a7-b5f0-30f36e377202-oauth-serving-cert\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.566975 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.566945 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d48bff-9a51-48a7-b5f0-30f36e377202-service-ca\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.567157 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.567140 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9d48bff-9a51-48a7-b5f0-30f36e377202-oauth-serving-cert\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.567378 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.567355 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9d48bff-9a51-48a7-b5f0-30f36e377202-console-config\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.567581 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.567564 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d48bff-9a51-48a7-b5f0-30f36e377202-trusted-ca-bundle\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.568909 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.568882 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9d48bff-9a51-48a7-b5f0-30f36e377202-console-oauth-config\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.569191 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.569174 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9d48bff-9a51-48a7-b5f0-30f36e377202-console-serving-cert\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.577425 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.577407 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfjvg\" (UniqueName: \"kubernetes.io/projected/a9d48bff-9a51-48a7-b5f0-30f36e377202-kube-api-access-dfjvg\") pod \"console-686ff6c96c-fkk4w\" (UID: \"a9d48bff-9a51-48a7-b5f0-30f36e377202\") " pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.577622 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.577605 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwj6r\" (UniqueName: \"kubernetes.io/projected/b9698f39-22cb-4ab7-b5ac-7b88b7597d49-kube-api-access-bwj6r\") pod \"kuadrant-operator-catalog-8zmqt\" (UID: \"b9698f39-22cb-4ab7-b5ac-7b88b7597d49\") " pod="kuadrant-system/kuadrant-operator-catalog-8zmqt" Jul 06 15:34:10.580496 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.580477 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:10.628341 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.628269 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8zmqt"] Jul 06 15:34:10.628638 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.628622 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8zmqt" Jul 06 15:34:10.734942 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.734917 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-686ff6c96c-fkk4w"] Jul 06 15:34:10.736976 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:34:10.736935 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d48bff_9a51_48a7_b5f0_30f36e377202.slice/crio-c2ebb21d3f0dbf412abe04972cd2e23f01f7d92f24d86d425bed25037a4f2508 WatchSource:0}: Error finding container c2ebb21d3f0dbf412abe04972cd2e23f01f7d92f24d86d425bed25037a4f2508: Status 404 returned error can't find the container with id c2ebb21d3f0dbf412abe04972cd2e23f01f7d92f24d86d425bed25037a4f2508 Jul 06 15:34:10.791293 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.791269 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8zmqt"] Jul 06 15:34:10.792450 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:34:10.792418 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9698f39_22cb_4ab7_b5ac_7b88b7597d49.slice/crio-d59bc0e30c657359537ee2146c4af277be38a0a403193d0a6abc0b94b5aded77 WatchSource:0}: Error finding container d59bc0e30c657359537ee2146c4af277be38a0a403193d0a6abc0b94b5aded77: Status 404 returned error can't find the container with id d59bc0e30c657359537ee2146c4af277be38a0a403193d0a6abc0b94b5aded77 Jul 06 15:34:10.810143 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.810112 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8zmqt" event={"ID":"b9698f39-22cb-4ab7-b5ac-7b88b7597d49","Type":"ContainerStarted","Data":"d59bc0e30c657359537ee2146c4af277be38a0a403193d0a6abc0b94b5aded77"} Jul 06 15:34:10.811678 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.811655 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-686ff6c96c-fkk4w" event={"ID":"a9d48bff-9a51-48a7-b5f0-30f36e377202","Type":"ContainerStarted","Data":"8156cd6bf899ddecceea71f0df87a9c5c75ff46bc249df7b5852d0535b3ba8ff"} Jul 06 15:34:10.811789 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.811695 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-686ff6c96c-fkk4w" event={"ID":"a9d48bff-9a51-48a7-b5f0-30f36e377202","Type":"ContainerStarted","Data":"c2ebb21d3f0dbf412abe04972cd2e23f01f7d92f24d86d425bed25037a4f2508"} Jul 06 15:34:10.848405 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.848157 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-686ff6c96c-fkk4w" podStartSLOduration=0.848136931 podStartE2EDuration="848.136931ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:10.846167949 +0000 UTC m=+472.478719956" watchObservedRunningTime="2026-07-06 15:34:10.848136931 +0000 UTC m=+472.480688915" Jul 06 15:34:10.858795 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.858768 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vnlc6"] Jul 06 15:34:10.862280 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.862256 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vnlc6" Jul 06 15:34:10.877397 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.877367 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vnlc6"] Jul 06 15:34:10.970603 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:10.970511 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz68x\" (UniqueName: \"kubernetes.io/projected/c97007f0-45d4-42a8-b9b7-cab3728abaf0-kube-api-access-fz68x\") pod \"kuadrant-operator-catalog-vnlc6\" (UID: \"c97007f0-45d4-42a8-b9b7-cab3728abaf0\") " pod="kuadrant-system/kuadrant-operator-catalog-vnlc6" Jul 06 15:34:11.071237 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:11.071208 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz68x\" (UniqueName: \"kubernetes.io/projected/c97007f0-45d4-42a8-b9b7-cab3728abaf0-kube-api-access-fz68x\") pod \"kuadrant-operator-catalog-vnlc6\" (UID: \"c97007f0-45d4-42a8-b9b7-cab3728abaf0\") " pod="kuadrant-system/kuadrant-operator-catalog-vnlc6" Jul 06 15:34:11.080647 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:11.080627 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz68x\" (UniqueName: \"kubernetes.io/projected/c97007f0-45d4-42a8-b9b7-cab3728abaf0-kube-api-access-fz68x\") pod \"kuadrant-operator-catalog-vnlc6\" (UID: \"c97007f0-45d4-42a8-b9b7-cab3728abaf0\") " pod="kuadrant-system/kuadrant-operator-catalog-vnlc6" Jul 06 15:34:11.082877 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:11.082849 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:11.171280 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:11.171253 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vnlc6" Jul 06 15:34:11.292130 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:11.292099 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:11.311567 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:11.311544 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vnlc6"] Jul 06 15:34:11.313674 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:34:11.313647 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97007f0_45d4_42a8_b9b7_cab3728abaf0.slice/crio-cc476aa566de5c9f77a1ba30785d3188436a2f967952e2f6dd39be22c8296729 WatchSource:0}: Error finding container cc476aa566de5c9f77a1ba30785d3188436a2f967952e2f6dd39be22c8296729: Status 404 returned error can't find the container with id cc476aa566de5c9f77a1ba30785d3188436a2f967952e2f6dd39be22c8296729 Jul 06 15:34:11.819672 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:11.819639 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vnlc6" event={"ID":"c97007f0-45d4-42a8-b9b7-cab3728abaf0","Type":"ContainerStarted","Data":"cc476aa566de5c9f77a1ba30785d3188436a2f967952e2f6dd39be22c8296729"} Jul 06 15:34:13.830234 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:13.830194 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vnlc6" event={"ID":"c97007f0-45d4-42a8-b9b7-cab3728abaf0","Type":"ContainerStarted","Data":"7ca0aa2500c1835c64de8008d9edfe21bebe5415bc601bff18f4ef6f17d3b443"} Jul 06 15:34:13.831600 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:13.831573 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8zmqt" event={"ID":"b9698f39-22cb-4ab7-b5ac-7b88b7597d49","Type":"ContainerStarted","Data":"d3b51f19f749a0115cec05d516ae924018583c058df229f1f67d6bb099320eca"} Jul 06 15:34:13.831737 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:13.831685 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-8zmqt" podUID="b9698f39-22cb-4ab7-b5ac-7b88b7597d49" containerName="registry-server" containerID="cri-o://d3b51f19f749a0115cec05d516ae924018583c058df229f1f67d6bb099320eca" gracePeriod=2 Jul 06 15:34:13.851314 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:13.851277 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-vnlc6" podStartSLOduration=2.076610201 podStartE2EDuration="3.851262019s" totalImagesPullingTime="1.774651818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:13.849907084 +0000 UTC m=+475.482459066" watchObservedRunningTime="2026-07-06 15:34:13.851262019 +0000 UTC m=+475.483814005" Jul 06 15:34:13.867660 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:13.867622 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8zmqt" podStartSLOduration=1.574710738 podStartE2EDuration="3.867608758s" totalImagesPullingTime="2.29289802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:13.867309349 +0000 UTC m=+475.499861322" watchObservedRunningTime="2026-07-06 15:34:13.867608758 +0000 UTC m=+475.500160740" Jul 06 15:34:14.082239 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:14.082184 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8zmqt" Jul 06 15:34:14.091859 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:14.091836 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwj6r\" (UniqueName: \"kubernetes.io/projected/b9698f39-22cb-4ab7-b5ac-7b88b7597d49-kube-api-access-bwj6r\") pod \"b9698f39-22cb-4ab7-b5ac-7b88b7597d49\" (UID: \"b9698f39-22cb-4ab7-b5ac-7b88b7597d49\") " Jul 06 15:34:14.093909 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:14.093884 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9698f39-22cb-4ab7-b5ac-7b88b7597d49-kube-api-access-bwj6r" (OuterVolumeSpecName: "kube-api-access-bwj6r") pod "b9698f39-22cb-4ab7-b5ac-7b88b7597d49" (UID: "b9698f39-22cb-4ab7-b5ac-7b88b7597d49"). InnerVolumeSpecName "kube-api-access-bwj6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:34:14.192564 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:14.192517 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwj6r\" (UniqueName: \"kubernetes.io/projected/b9698f39-22cb-4ab7-b5ac-7b88b7597d49-kube-api-access-bwj6r\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:34:14.836799 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:14.836769 2561 generic.go:356] "Generic (PLEG): container finished" podID="b9698f39-22cb-4ab7-b5ac-7b88b7597d49" containerID="d3b51f19f749a0115cec05d516ae924018583c058df229f1f67d6bb099320eca" exitCode=0 Jul 06 15:34:14.837192 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:14.836830 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8zmqt" Jul 06 15:34:14.837192 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:14.836855 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8zmqt" event={"ID":"b9698f39-22cb-4ab7-b5ac-7b88b7597d49","Type":"ContainerDied","Data":"d3b51f19f749a0115cec05d516ae924018583c058df229f1f67d6bb099320eca"} Jul 06 15:34:14.837192 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:14.836910 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8zmqt" event={"ID":"b9698f39-22cb-4ab7-b5ac-7b88b7597d49","Type":"ContainerDied","Data":"d59bc0e30c657359537ee2146c4af277be38a0a403193d0a6abc0b94b5aded77"} Jul 06 15:34:14.837192 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:14.836930 2561 scope.go:117] "RemoveContainer" containerID="d3b51f19f749a0115cec05d516ae924018583c058df229f1f67d6bb099320eca" Jul 06 15:34:14.845992 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:14.845980 2561 scope.go:117] "RemoveContainer" containerID="d3b51f19f749a0115cec05d516ae924018583c058df229f1f67d6bb099320eca" Jul 06 15:34:14.846241 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:34:14.846225 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b51f19f749a0115cec05d516ae924018583c058df229f1f67d6bb099320eca\": container with ID starting with d3b51f19f749a0115cec05d516ae924018583c058df229f1f67d6bb099320eca not found: ID does not exist" containerID="d3b51f19f749a0115cec05d516ae924018583c058df229f1f67d6bb099320eca" Jul 06 15:34:14.846355 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:14.846249 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b51f19f749a0115cec05d516ae924018583c058df229f1f67d6bb099320eca"} err="failed to get container status \"d3b51f19f749a0115cec05d516ae924018583c058df229f1f67d6bb099320eca\": rpc error: code = NotFound desc = could not find container \"d3b51f19f749a0115cec05d516ae924018583c058df229f1f67d6bb099320eca\": container with ID starting with d3b51f19f749a0115cec05d516ae924018583c058df229f1f67d6bb099320eca not found: ID does not exist" Jul 06 15:34:14.857852 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:14.857833 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8zmqt"] Jul 06 15:34:14.863045 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:14.863024 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8zmqt"] Jul 06 15:34:16.799637 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:16.799602 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9698f39-22cb-4ab7-b5ac-7b88b7597d49" path="/var/lib/kubelet/pods/b9698f39-22cb-4ab7-b5ac-7b88b7597d49/volumes" Jul 06 15:34:20.581612 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:20.581577 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:20.582016 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:20.581684 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:20.586409 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:20.586386 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:20.868679 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:20.868606 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-686ff6c96c-fkk4w" Jul 06 15:34:21.054284 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:21.054242 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:21.151884 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:21.151805 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:21.171369 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:21.171343 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-vnlc6" Jul 06 15:34:21.171493 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:21.171398 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-vnlc6" Jul 06 15:34:21.194232 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:21.194202 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-vnlc6" Jul 06 15:34:21.226683 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:21.226660 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:21.891092 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:21.891067 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-vnlc6" Jul 06 15:34:40.227805 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:40.227763 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:43.750824 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:43.750774 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:43.866918 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:43.866880 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:47.945437 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:47.945405 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2"] Jul 06 15:34:47.945849 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:47.945835 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9698f39-22cb-4ab7-b5ac-7b88b7597d49" containerName="registry-server" Jul 06 15:34:47.945892 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:47.945852 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9698f39-22cb-4ab7-b5ac-7b88b7597d49" containerName="registry-server" Jul 06 15:34:47.945931 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:47.945918 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9698f39-22cb-4ab7-b5ac-7b88b7597d49" containerName="registry-server" Jul 06 15:34:47.949927 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:47.949913 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:34:47.953150 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:47.953122 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-99tx8\"" Jul 06 15:34:47.962022 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:47.962001 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2"] Jul 06 15:34:48.075204 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:48.075178 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvm5d\" (UniqueName: \"kubernetes.io/projected/ece6cf5a-8a7a-4b44-812e-470b7d062ab6-kube-api-access-kvm5d\") pod \"kuadrant-operator-controller-manager-8684c9ff49-26nb2\" (UID: \"ece6cf5a-8a7a-4b44-812e-470b7d062ab6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:34:48.075358 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:48.075219 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ece6cf5a-8a7a-4b44-812e-470b7d062ab6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-26nb2\" (UID: \"ece6cf5a-8a7a-4b44-812e-470b7d062ab6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:34:48.176237 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:48.176207 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvm5d\" (UniqueName: \"kubernetes.io/projected/ece6cf5a-8a7a-4b44-812e-470b7d062ab6-kube-api-access-kvm5d\") pod \"kuadrant-operator-controller-manager-8684c9ff49-26nb2\" (UID: \"ece6cf5a-8a7a-4b44-812e-470b7d062ab6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:34:48.176359 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:48.176246 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ece6cf5a-8a7a-4b44-812e-470b7d062ab6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-26nb2\" (UID: \"ece6cf5a-8a7a-4b44-812e-470b7d062ab6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:34:48.176702 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:48.176686 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ece6cf5a-8a7a-4b44-812e-470b7d062ab6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-26nb2\" (UID: \"ece6cf5a-8a7a-4b44-812e-470b7d062ab6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:34:48.192024 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:48.191992 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvm5d\" (UniqueName: \"kubernetes.io/projected/ece6cf5a-8a7a-4b44-812e-470b7d062ab6-kube-api-access-kvm5d\") pod \"kuadrant-operator-controller-manager-8684c9ff49-26nb2\" (UID: \"ece6cf5a-8a7a-4b44-812e-470b7d062ab6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:34:48.260260 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:48.260231 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:34:48.353140 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:48.353107 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:48.404485 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:48.404462 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2"] Jul 06 15:34:48.406354 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:34:48.406316 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece6cf5a_8a7a_4b44_812e_470b7d062ab6.slice/crio-3fd20e0fad3cdd9d0d7c2c7a17a8a57fdf2f75b70bcab3e48568c1a45c7fb639 WatchSource:0}: Error finding container 3fd20e0fad3cdd9d0d7c2c7a17a8a57fdf2f75b70bcab3e48568c1a45c7fb639: Status 404 returned error can't find the container with id 3fd20e0fad3cdd9d0d7c2c7a17a8a57fdf2f75b70bcab3e48568c1a45c7fb639 Jul 06 15:34:48.780659 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:48.780632 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:48.996896 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:48.996865 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" event={"ID":"ece6cf5a-8a7a-4b44-812e-470b7d062ab6","Type":"ContainerStarted","Data":"3fd20e0fad3cdd9d0d7c2c7a17a8a57fdf2f75b70bcab3e48568c1a45c7fb639"} Jul 06 15:34:50.179552 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:50.179503 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:51.776749 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:51.776712 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:52.598642 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:52.598610 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:52.693661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:52.693629 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:54.080772 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:54.080745 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4kkt"] Jul 06 15:34:54.084128 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:54.084112 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4kkt" Jul 06 15:34:54.088684 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:54.088662 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 15:34:54.088783 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:54.088724 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-tlcnt\"" Jul 06 15:34:54.101822 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:54.101797 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4kkt"] Jul 06 15:34:54.228459 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:54.228428 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gpmz\" (UniqueName: \"kubernetes.io/projected/3b1bc0d4-54a1-4658-9597-1451b967cdea-kube-api-access-8gpmz\") pod \"dns-operator-controller-manager-864dd78c4c-z4kkt\" (UID: \"3b1bc0d4-54a1-4658-9597-1451b967cdea\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4kkt" Jul 06 15:34:54.329557 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:54.329504 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gpmz\" (UniqueName: \"kubernetes.io/projected/3b1bc0d4-54a1-4658-9597-1451b967cdea-kube-api-access-8gpmz\") pod \"dns-operator-controller-manager-864dd78c4c-z4kkt\" (UID: \"3b1bc0d4-54a1-4658-9597-1451b967cdea\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4kkt" Jul 06 15:34:54.348002 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:54.347937 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gpmz\" (UniqueName: \"kubernetes.io/projected/3b1bc0d4-54a1-4658-9597-1451b967cdea-kube-api-access-8gpmz\") pod \"dns-operator-controller-manager-864dd78c4c-z4kkt\" (UID: \"3b1bc0d4-54a1-4658-9597-1451b967cdea\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4kkt" Jul 06 15:34:54.394356 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:54.394328 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4kkt" Jul 06 15:34:54.692166 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:54.692084 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:55.680859 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:55.680834 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4kkt"] Jul 06 15:34:55.681910 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:34:55.681887 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1bc0d4_54a1_4658_9597_1451b967cdea.slice/crio-f5524c774e7495bac6d1d57886aeade954c3bd83f0bf0de4ab7ad3aafb1b30bb WatchSource:0}: Error finding container f5524c774e7495bac6d1d57886aeade954c3bd83f0bf0de4ab7ad3aafb1b30bb: Status 404 returned error can't find the container with id f5524c774e7495bac6d1d57886aeade954c3bd83f0bf0de4ab7ad3aafb1b30bb Jul 06 15:34:56.031816 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:56.031749 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" event={"ID":"ece6cf5a-8a7a-4b44-812e-470b7d062ab6","Type":"ContainerStarted","Data":"f3a65c6787f453f67742ed7b38a697cfd97e634772e91009069659f00da5e0a7"} Jul 06 15:34:56.032003 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:56.031835 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:34:56.033143 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:56.033111 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4kkt" event={"ID":"3b1bc0d4-54a1-4658-9597-1451b967cdea","Type":"ContainerStarted","Data":"f5524c774e7495bac6d1d57886aeade954c3bd83f0bf0de4ab7ad3aafb1b30bb"} Jul 06 15:34:56.068140 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:56.068096 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:56.083598 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:56.083562 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" podStartSLOduration=1.9328001430000001 podStartE2EDuration="9.083547763s" totalImagesPullingTime="7.15074762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:56.082621326 +0000 UTC m=+517.715173312" watchObservedRunningTime="2026-07-06 15:34:56.083547763 +0000 UTC m=+517.716099772" Jul 06 15:34:56.217331 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:56.217297 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:58.045064 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:58.045034 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4kkt" event={"ID":"3b1bc0d4-54a1-4658-9597-1451b967cdea","Type":"ContainerStarted","Data":"6c6fb70eb6a954e532e60cb0ded4b7884580c5ba38ca9a5e78e6d8f5ba20ad1f"} Jul 06 15:34:58.045433 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:58.045088 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4kkt" Jul 06 15:34:58.082402 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:58.082346 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4kkt" podStartSLOduration=1.911575389 podStartE2EDuration="4.082331435s" totalImagesPullingTime="2.170756046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:58.080432738 +0000 UTC m=+519.712984721" watchObservedRunningTime="2026-07-06 15:34:58.082331435 +0000 UTC m=+519.714883418" Jul 06 15:34:58.968208 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:58.968177 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:59.115510 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:59.115476 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:59.569973 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:59.569884 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:59.702441 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:34:59.702407 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:07.041349 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:07.041321 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:35:08.151723 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:08.151688 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:08.303429 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:08.303393 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:09.053818 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.053795 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-z4kkt" Jul 06 15:35:09.227622 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.227588 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2"] Jul 06 15:35:09.228043 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.227790 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" podUID="ece6cf5a-8a7a-4b44-812e-470b7d062ab6" containerName="manager" containerID="cri-o://f3a65c6787f453f67742ed7b38a697cfd97e634772e91009069659f00da5e0a7" gracePeriod=2 Jul 06 15:35:09.240042 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.240013 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2"] Jul 06 15:35:09.242182 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:35:09.242153 2561 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ece6cf5a-8a7a-4b44-812e-470b7d062ab6\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-26nb2\": pods \"kuadrant-operator-controller-manager-8684c9ff49-26nb2\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:35:09.267474 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.267453 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p"] Jul 06 15:35:09.267927 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.267909 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece6cf5a-8a7a-4b44-812e-470b7d062ab6" containerName="manager" Jul 06 15:35:09.268009 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.267930 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece6cf5a-8a7a-4b44-812e-470b7d062ab6" containerName="manager" Jul 06 15:35:09.268009 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.268001 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece6cf5a-8a7a-4b44-812e-470b7d062ab6" containerName="manager" Jul 06 15:35:09.271051 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.271034 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" Jul 06 15:35:09.273952 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:35:09.273927 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-26nb2\" is forbidden: User \"system:node:ip-10-0-129-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-225.ec2.internal' and this object" podUID="ece6cf5a-8a7a-4b44-812e-470b7d062ab6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:35:09.286734 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.286710 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p"] Jul 06 15:35:09.349995 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.349968 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9763f745-0b9f-4646-8916-20b0d6a5789c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-77z2p\" (UID: \"9763f745-0b9f-4646-8916-20b0d6a5789c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" Jul 06 15:35:09.350098 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.350053 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk9nv\" (UniqueName: \"kubernetes.io/projected/9763f745-0b9f-4646-8916-20b0d6a5789c-kube-api-access-qk9nv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-77z2p\" (UID: \"9763f745-0b9f-4646-8916-20b0d6a5789c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" Jul 06 15:35:09.450912 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.450886 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9763f745-0b9f-4646-8916-20b0d6a5789c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-77z2p\" (UID: \"9763f745-0b9f-4646-8916-20b0d6a5789c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" Jul 06 15:35:09.451030 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.450969 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk9nv\" (UniqueName: \"kubernetes.io/projected/9763f745-0b9f-4646-8916-20b0d6a5789c-kube-api-access-qk9nv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-77z2p\" (UID: \"9763f745-0b9f-4646-8916-20b0d6a5789c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" Jul 06 15:35:09.451327 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.451303 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9763f745-0b9f-4646-8916-20b0d6a5789c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-77z2p\" (UID: \"9763f745-0b9f-4646-8916-20b0d6a5789c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" Jul 06 15:35:09.462650 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.462623 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk9nv\" (UniqueName: \"kubernetes.io/projected/9763f745-0b9f-4646-8916-20b0d6a5789c-kube-api-access-qk9nv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-77z2p\" (UID: \"9763f745-0b9f-4646-8916-20b0d6a5789c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" Jul 06 15:35:09.473906 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.473881 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:35:09.476864 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:35:09.476844 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-26nb2\" is forbidden: User \"system:node:ip-10-0-129-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-225.ec2.internal' and this object" podUID="ece6cf5a-8a7a-4b44-812e-470b7d062ab6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:35:09.551234 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.551211 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ece6cf5a-8a7a-4b44-812e-470b7d062ab6-extensions-socket-volume\") pod \"ece6cf5a-8a7a-4b44-812e-470b7d062ab6\" (UID: \"ece6cf5a-8a7a-4b44-812e-470b7d062ab6\") " Jul 06 15:35:09.551328 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.551280 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvm5d\" (UniqueName: \"kubernetes.io/projected/ece6cf5a-8a7a-4b44-812e-470b7d062ab6-kube-api-access-kvm5d\") pod \"ece6cf5a-8a7a-4b44-812e-470b7d062ab6\" (UID: \"ece6cf5a-8a7a-4b44-812e-470b7d062ab6\") " Jul 06 15:35:09.551761 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.551733 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ece6cf5a-8a7a-4b44-812e-470b7d062ab6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ece6cf5a-8a7a-4b44-812e-470b7d062ab6" (UID: "ece6cf5a-8a7a-4b44-812e-470b7d062ab6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:35:09.553301 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.553280 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece6cf5a-8a7a-4b44-812e-470b7d062ab6-kube-api-access-kvm5d" (OuterVolumeSpecName: "kube-api-access-kvm5d") pod "ece6cf5a-8a7a-4b44-812e-470b7d062ab6" (UID: "ece6cf5a-8a7a-4b44-812e-470b7d062ab6"). InnerVolumeSpecName "kube-api-access-kvm5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:35:09.609499 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.609475 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" Jul 06 15:35:09.652348 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.652318 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ece6cf5a-8a7a-4b44-812e-470b7d062ab6-extensions-socket-volume\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:35:09.652348 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.652345 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvm5d\" (UniqueName: \"kubernetes.io/projected/ece6cf5a-8a7a-4b44-812e-470b7d062ab6-kube-api-access-kvm5d\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:35:09.668557 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.668508 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd"] Jul 06 15:35:09.673889 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.673866 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" Jul 06 15:35:09.704606 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.704579 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd"] Jul 06 15:35:09.730416 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:35:09.730385 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-26nb2\" is forbidden: User \"system:node:ip-10-0-129-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-225.ec2.internal' and this object" podUID="ece6cf5a-8a7a-4b44-812e-470b7d062ab6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:35:09.752972 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.752943 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tftt\" (UniqueName: \"kubernetes.io/projected/85d66866-ed02-4e93-a6d5-a6ca57a6b349-kube-api-access-7tftt\") pod \"kuadrant-operator-controller-manager-86878c769f-fnpqd\" (UID: \"85d66866-ed02-4e93-a6d5-a6ca57a6b349\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" Jul 06 15:35:09.753080 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.752981 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85d66866-ed02-4e93-a6d5-a6ca57a6b349-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fnpqd\" (UID: \"85d66866-ed02-4e93-a6d5-a6ca57a6b349\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" Jul 06 15:35:09.779509 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.779489 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p"] Jul 06 15:35:09.781974 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:35:09.781952 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9763f745_0b9f_4646_8916_20b0d6a5789c.slice/crio-f2176fca88ef65419fadd3b26cdd0b2bb275eb8e3cb65a1e69e292f97a656ff5 WatchSource:0}: Error finding container f2176fca88ef65419fadd3b26cdd0b2bb275eb8e3cb65a1e69e292f97a656ff5: Status 404 returned error can't find the container with id f2176fca88ef65419fadd3b26cdd0b2bb275eb8e3cb65a1e69e292f97a656ff5 Jul 06 15:35:09.853848 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.853824 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tftt\" (UniqueName: \"kubernetes.io/projected/85d66866-ed02-4e93-a6d5-a6ca57a6b349-kube-api-access-7tftt\") pod \"kuadrant-operator-controller-manager-86878c769f-fnpqd\" (UID: \"85d66866-ed02-4e93-a6d5-a6ca57a6b349\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" Jul 06 15:35:09.853940 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.853901 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85d66866-ed02-4e93-a6d5-a6ca57a6b349-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fnpqd\" (UID: \"85d66866-ed02-4e93-a6d5-a6ca57a6b349\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" Jul 06 15:35:09.854236 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.854221 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85d66866-ed02-4e93-a6d5-a6ca57a6b349-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fnpqd\" (UID: \"85d66866-ed02-4e93-a6d5-a6ca57a6b349\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" Jul 06 15:35:09.873087 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.873064 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tftt\" (UniqueName: \"kubernetes.io/projected/85d66866-ed02-4e93-a6d5-a6ca57a6b349-kube-api-access-7tftt\") pod \"kuadrant-operator-controller-manager-86878c769f-fnpqd\" (UID: \"85d66866-ed02-4e93-a6d5-a6ca57a6b349\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" Jul 06 15:35:09.984660 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:09.984633 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" Jul 06 15:35:10.102904 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:10.102878 2561 generic.go:356] "Generic (PLEG): container finished" podID="ece6cf5a-8a7a-4b44-812e-470b7d062ab6" containerID="f3a65c6787f453f67742ed7b38a697cfd97e634772e91009069659f00da5e0a7" exitCode=0 Jul 06 15:35:10.103070 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:10.102917 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:35:10.103070 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:10.102985 2561 scope.go:117] "RemoveContainer" containerID="f3a65c6787f453f67742ed7b38a697cfd97e634772e91009069659f00da5e0a7" Jul 06 15:35:10.104755 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:10.104726 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" event={"ID":"9763f745-0b9f-4646-8916-20b0d6a5789c","Type":"ContainerStarted","Data":"7911dae96776426689f5fb8f41389a711b4d4ed2d55b8cb8fc4a84ff9dcf08d9"} Jul 06 15:35:10.104891 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:10.104780 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" event={"ID":"9763f745-0b9f-4646-8916-20b0d6a5789c","Type":"ContainerStarted","Data":"f2176fca88ef65419fadd3b26cdd0b2bb275eb8e3cb65a1e69e292f97a656ff5"} Jul 06 15:35:10.104968 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:10.104897 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" Jul 06 15:35:10.106638 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:35:10.106615 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-26nb2\" is forbidden: User \"system:node:ip-10-0-129-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-225.ec2.internal' and this object" podUID="ece6cf5a-8a7a-4b44-812e-470b7d062ab6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:35:10.109724 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:35:10.109690 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-26nb2\" is forbidden: User \"system:node:ip-10-0-129-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-225.ec2.internal' and this object" podUID="ece6cf5a-8a7a-4b44-812e-470b7d062ab6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:35:10.112758 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:10.112742 2561 scope.go:117] "RemoveContainer" containerID="f3a65c6787f453f67742ed7b38a697cfd97e634772e91009069659f00da5e0a7" Jul 06 15:35:10.113004 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:35:10.112983 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a65c6787f453f67742ed7b38a697cfd97e634772e91009069659f00da5e0a7\": container with ID starting with f3a65c6787f453f67742ed7b38a697cfd97e634772e91009069659f00da5e0a7 not found: ID does not exist" containerID="f3a65c6787f453f67742ed7b38a697cfd97e634772e91009069659f00da5e0a7" Jul 06 15:35:10.113045 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:10.113013 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a65c6787f453f67742ed7b38a697cfd97e634772e91009069659f00da5e0a7"} err="failed to get container status \"f3a65c6787f453f67742ed7b38a697cfd97e634772e91009069659f00da5e0a7\": rpc error: code = NotFound desc = could not find container \"f3a65c6787f453f67742ed7b38a697cfd97e634772e91009069659f00da5e0a7\": container with ID starting with f3a65c6787f453f67742ed7b38a697cfd97e634772e91009069659f00da5e0a7 not found: ID does not exist" Jul 06 15:35:10.129774 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:10.129756 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd"] Jul 06 15:35:10.132417 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:35:10.132391 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d66866_ed02_4e93_a6d5_a6ca57a6b349.slice/crio-cdf6b6569bbd6e6e2566f41c4299a3e8604455f827141b41c8e7d1af0126ff6a WatchSource:0}: Error finding container cdf6b6569bbd6e6e2566f41c4299a3e8604455f827141b41c8e7d1af0126ff6a: Status 404 returned error can't find the container with id cdf6b6569bbd6e6e2566f41c4299a3e8604455f827141b41c8e7d1af0126ff6a Jul 06 15:35:10.149473 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:10.149432 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" podStartSLOduration=1.14941952 podStartE2EDuration="1.14941952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:10.14854392 +0000 UTC m=+531.781095899" watchObservedRunningTime="2026-07-06 15:35:10.14941952 +0000 UTC m=+531.781971503" Jul 06 15:35:10.151043 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:35:10.151020 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-26nb2\" is forbidden: User \"system:node:ip-10-0-129-225.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-225.ec2.internal' and this object" podUID="ece6cf5a-8a7a-4b44-812e-470b7d062ab6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-26nb2" Jul 06 15:35:10.565209 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:10.565168 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:10.682326 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:10.682293 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:10.806352 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:10.806307 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ece6cf5a-8a7a-4b44-812e-470b7d062ab6" path="/var/lib/kubelet/pods/ece6cf5a-8a7a-4b44-812e-470b7d062ab6/volumes" Jul 06 15:35:11.114252 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:11.114209 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" event={"ID":"85d66866-ed02-4e93-a6d5-a6ca57a6b349","Type":"ContainerStarted","Data":"e8dea3319bac8025ce962be604dcf1ccb8ff7a5d726b34505ed41b367c1bcb89"} Jul 06 15:35:11.114429 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:11.114269 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" event={"ID":"85d66866-ed02-4e93-a6d5-a6ca57a6b349","Type":"ContainerStarted","Data":"cdf6b6569bbd6e6e2566f41c4299a3e8604455f827141b41c8e7d1af0126ff6a"} Jul 06 15:35:11.145092 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:11.145041 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" podStartSLOduration=2.145021687 podStartE2EDuration="2.145021687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:11.144500889 +0000 UTC m=+532.777052871" watchObservedRunningTime="2026-07-06 15:35:11.145021687 +0000 UTC m=+532.777573670" Jul 06 15:35:12.118331 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:12.118306 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" Jul 06 15:35:14.769067 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:14.769039 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:14.867291 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:14.867257 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:17.017461 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:17.017422 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:17.178212 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:17.178170 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:21.116931 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:21.116898 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" Jul 06 15:35:23.125446 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:23.125416 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" Jul 06 15:35:23.218976 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:23.218944 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p"] Jul 06 15:35:23.219150 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:23.219118 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" podUID="9763f745-0b9f-4646-8916-20b0d6a5789c" containerName="manager" containerID="cri-o://7911dae96776426689f5fb8f41389a711b4d4ed2d55b8cb8fc4a84ff9dcf08d9" gracePeriod=10 Jul 06 15:35:23.466249 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:23.466227 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" Jul 06 15:35:23.571992 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:23.571968 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9763f745-0b9f-4646-8916-20b0d6a5789c-extensions-socket-volume\") pod \"9763f745-0b9f-4646-8916-20b0d6a5789c\" (UID: \"9763f745-0b9f-4646-8916-20b0d6a5789c\") " Jul 06 15:35:23.572115 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:23.572008 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk9nv\" (UniqueName: \"kubernetes.io/projected/9763f745-0b9f-4646-8916-20b0d6a5789c-kube-api-access-qk9nv\") pod \"9763f745-0b9f-4646-8916-20b0d6a5789c\" (UID: \"9763f745-0b9f-4646-8916-20b0d6a5789c\") " Jul 06 15:35:23.572405 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:23.572380 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9763f745-0b9f-4646-8916-20b0d6a5789c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9763f745-0b9f-4646-8916-20b0d6a5789c" (UID: "9763f745-0b9f-4646-8916-20b0d6a5789c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:35:23.574207 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:23.574181 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9763f745-0b9f-4646-8916-20b0d6a5789c-kube-api-access-qk9nv" (OuterVolumeSpecName: "kube-api-access-qk9nv") pod "9763f745-0b9f-4646-8916-20b0d6a5789c" (UID: "9763f745-0b9f-4646-8916-20b0d6a5789c"). InnerVolumeSpecName "kube-api-access-qk9nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:35:23.673058 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:23.672994 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk9nv\" (UniqueName: \"kubernetes.io/projected/9763f745-0b9f-4646-8916-20b0d6a5789c-kube-api-access-qk9nv\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:35:23.673058 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:23.673018 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9763f745-0b9f-4646-8916-20b0d6a5789c-extensions-socket-volume\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:35:24.175041 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:24.175013 2561 generic.go:356] "Generic (PLEG): container finished" podID="9763f745-0b9f-4646-8916-20b0d6a5789c" containerID="7911dae96776426689f5fb8f41389a711b4d4ed2d55b8cb8fc4a84ff9dcf08d9" exitCode=0 Jul 06 15:35:24.175415 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:24.175056 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" Jul 06 15:35:24.175415 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:24.175100 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" event={"ID":"9763f745-0b9f-4646-8916-20b0d6a5789c","Type":"ContainerDied","Data":"7911dae96776426689f5fb8f41389a711b4d4ed2d55b8cb8fc4a84ff9dcf08d9"} Jul 06 15:35:24.175415 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:24.175149 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p" event={"ID":"9763f745-0b9f-4646-8916-20b0d6a5789c","Type":"ContainerDied","Data":"f2176fca88ef65419fadd3b26cdd0b2bb275eb8e3cb65a1e69e292f97a656ff5"} Jul 06 15:35:24.175415 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:24.175173 2561 scope.go:117] "RemoveContainer" containerID="7911dae96776426689f5fb8f41389a711b4d4ed2d55b8cb8fc4a84ff9dcf08d9" Jul 06 15:35:24.184398 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:24.184382 2561 scope.go:117] "RemoveContainer" containerID="7911dae96776426689f5fb8f41389a711b4d4ed2d55b8cb8fc4a84ff9dcf08d9" Jul 06 15:35:24.184679 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:35:24.184655 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7911dae96776426689f5fb8f41389a711b4d4ed2d55b8cb8fc4a84ff9dcf08d9\": container with ID starting with 7911dae96776426689f5fb8f41389a711b4d4ed2d55b8cb8fc4a84ff9dcf08d9 not found: ID does not exist" containerID="7911dae96776426689f5fb8f41389a711b4d4ed2d55b8cb8fc4a84ff9dcf08d9" Jul 06 15:35:24.184909 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:24.184686 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7911dae96776426689f5fb8f41389a711b4d4ed2d55b8cb8fc4a84ff9dcf08d9"} err="failed to get container status \"7911dae96776426689f5fb8f41389a711b4d4ed2d55b8cb8fc4a84ff9dcf08d9\": rpc error: code = NotFound desc = could not find container \"7911dae96776426689f5fb8f41389a711b4d4ed2d55b8cb8fc4a84ff9dcf08d9\": container with ID starting with 7911dae96776426689f5fb8f41389a711b4d4ed2d55b8cb8fc4a84ff9dcf08d9 not found: ID does not exist" Jul 06 15:35:24.205599 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:24.205580 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p"] Jul 06 15:35:24.214462 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:24.214442 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-77z2p"] Jul 06 15:35:24.804366 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:24.804324 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9763f745-0b9f-4646-8916-20b0d6a5789c" path="/var/lib/kubelet/pods/9763f745-0b9f-4646-8916-20b0d6a5789c/volumes" Jul 06 15:35:25.845269 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:25.845235 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:25.944318 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:25.944284 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:28.264472 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:28.264436 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:28.366381 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:28.366352 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:28.693390 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:28.693306 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:31.551547 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:31.551494 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:31.644763 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:31.644724 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:32.608415 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:32.608376 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:32.721262 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:32.721229 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:36.218967 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:36.218925 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:36.321943 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:36.321907 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:39.839902 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.839863 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf"] Jul 06 15:35:39.840353 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.840298 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9763f745-0b9f-4646-8916-20b0d6a5789c" containerName="manager" Jul 06 15:35:39.840353 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.840317 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9763f745-0b9f-4646-8916-20b0d6a5789c" containerName="manager" Jul 06 15:35:39.840476 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.840424 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9763f745-0b9f-4646-8916-20b0d6a5789c" containerName="manager" Jul 06 15:35:39.843713 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.843697 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:39.849604 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.849580 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zvfcv\"" Jul 06 15:35:39.861068 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.861049 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf"] Jul 06 15:35:39.898481 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.898448 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:39.916144 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.916122 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phphn\" (UniqueName: \"kubernetes.io/projected/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-kube-api-access-phphn\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:39.916268 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.916153 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:39.916268 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.916176 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:39.916341 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.916312 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:39.916384 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.916348 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:39.916384 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.916376 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:39.916456 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.916441 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:39.916655 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.916471 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:39.916987 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:39.916964 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.018116 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.018083 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.018116 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.018119 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.018332 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.018136 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.018332 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.018183 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.018332 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.018213 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.018332 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.018245 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.018575 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.018397 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phphn\" (UniqueName: \"kubernetes.io/projected/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-kube-api-access-phphn\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.018575 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.018443 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.018575 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.018471 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.018758 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.018578 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.018758 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.018622 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.018758 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.018640 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.018758 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.018689 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.019065 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.019041 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.021123 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.021101 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.021205 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.021143 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.029022 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.029001 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phphn\" (UniqueName: \"kubernetes.io/projected/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-kube-api-access-phphn\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.029292 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.029273 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9acfb2a-5f27-4ead-8402-a7a9d253d5ed-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-5nsgf\" (UID: \"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.108037 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.107968 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:40.153994 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.153968 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:40.297606 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.297580 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf"] Jul 06 15:35:40.299049 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:35:40.299025 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9acfb2a_5f27_4ead_8402_a7a9d253d5ed.slice/crio-17825262e30b614d6a46a6e2c4b6b741bba704a080e583d2b5db34367f762116 WatchSource:0}: Error finding container 17825262e30b614d6a46a6e2c4b6b741bba704a080e583d2b5db34367f762116: Status 404 returned error can't find the container with id 17825262e30b614d6a46a6e2c4b6b741bba704a080e583d2b5db34367f762116 Jul 06 15:35:40.301028 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.301000 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:35:40.301095 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.301060 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:35:40.301095 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:40.301087 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:35:41.257841 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:41.257811 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" event={"ID":"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed","Type":"ContainerStarted","Data":"1701f518e84b4b3c25462cd26338b7318b3df658c88df31275078f3dcb60e92a"} Jul 06 15:35:41.258211 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:41.257859 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" event={"ID":"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed","Type":"ContainerStarted","Data":"17825262e30b614d6a46a6e2c4b6b741bba704a080e583d2b5db34367f762116"} Jul 06 15:35:41.282112 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:41.282055 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" podStartSLOduration=2.282021547 podStartE2EDuration="2.282021547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:41.281110202 +0000 UTC m=+562.913662198" watchObservedRunningTime="2026-07-06 15:35:41.282021547 +0000 UTC m=+562.914573533" Jul 06 15:35:42.155150 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.155119 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:42.159760 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.159737 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:42.263937 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.263906 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:42.264907 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.264883 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:35:42.497030 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.497001 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:42.555013 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.554978 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2ccg9"] Jul 06 15:35:42.558645 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.558624 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" Jul 06 15:35:42.566104 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.566077 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 15:35:42.566197 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.566178 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6q2bl\"" Jul 06 15:35:42.575118 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.575093 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2ccg9"] Jul 06 15:35:42.596882 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.596848 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2ccg9"] Jul 06 15:35:42.615785 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.615734 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:42.641939 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.641914 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbhzg\" (UniqueName: \"kubernetes.io/projected/d2b17cf8-07dc-4aff-93ce-158dde108d2c-kube-api-access-jbhzg\") pod \"limitador-limitador-64c94fd64f-2ccg9\" (UID: \"d2b17cf8-07dc-4aff-93ce-158dde108d2c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" Jul 06 15:35:42.642064 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.642041 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2b17cf8-07dc-4aff-93ce-158dde108d2c-config-file\") pod \"limitador-limitador-64c94fd64f-2ccg9\" (UID: \"d2b17cf8-07dc-4aff-93ce-158dde108d2c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" Jul 06 15:35:42.692217 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.692192 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:42.743209 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.743181 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbhzg\" (UniqueName: \"kubernetes.io/projected/d2b17cf8-07dc-4aff-93ce-158dde108d2c-kube-api-access-jbhzg\") pod \"limitador-limitador-64c94fd64f-2ccg9\" (UID: \"d2b17cf8-07dc-4aff-93ce-158dde108d2c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" Jul 06 15:35:42.743352 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.743246 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2b17cf8-07dc-4aff-93ce-158dde108d2c-config-file\") pod \"limitador-limitador-64c94fd64f-2ccg9\" (UID: \"d2b17cf8-07dc-4aff-93ce-158dde108d2c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" Jul 06 15:35:42.743857 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.743841 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2b17cf8-07dc-4aff-93ce-158dde108d2c-config-file\") pod \"limitador-limitador-64c94fd64f-2ccg9\" (UID: \"d2b17cf8-07dc-4aff-93ce-158dde108d2c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" Jul 06 15:35:42.756423 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.756364 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbhzg\" (UniqueName: \"kubernetes.io/projected/d2b17cf8-07dc-4aff-93ce-158dde108d2c-kube-api-access-jbhzg\") pod \"limitador-limitador-64c94fd64f-2ccg9\" (UID: \"d2b17cf8-07dc-4aff-93ce-158dde108d2c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" Jul 06 15:35:42.869052 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:42.869024 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" Jul 06 15:35:43.006314 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:43.006286 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2ccg9"] Jul 06 15:35:43.007742 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:35:43.007686 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b17cf8_07dc_4aff_93ce_158dde108d2c.slice/crio-b691a50ef60f3726d3329a54d9dd0bbfe80ce1d925aeb21931f46d4cb8b8d3a5 WatchSource:0}: Error finding container b691a50ef60f3726d3329a54d9dd0bbfe80ce1d925aeb21931f46d4cb8b8d3a5: Status 404 returned error can't find the container with id b691a50ef60f3726d3329a54d9dd0bbfe80ce1d925aeb21931f46d4cb8b8d3a5 Jul 06 15:35:43.269200 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:43.269124 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" event={"ID":"d2b17cf8-07dc-4aff-93ce-158dde108d2c","Type":"ContainerStarted","Data":"b691a50ef60f3726d3329a54d9dd0bbfe80ce1d925aeb21931f46d4cb8b8d3a5"} Jul 06 15:35:43.694185 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:43.694100 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:43.815187 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:43.815157 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:46.285808 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:46.285774 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" event={"ID":"d2b17cf8-07dc-4aff-93ce-158dde108d2c","Type":"ContainerStarted","Data":"6c21266db7e3404d052acf2863f255449bf0dbec3aa03384b1b296ee4e9bdc3d"} Jul 06 15:35:46.286149 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:46.285861 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" Jul 06 15:35:46.307683 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:46.307638 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" podStartSLOduration=1.436501609 podStartE2EDuration="4.307624627s" totalImagesPullingTime="2.871123018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:46.306475427 +0000 UTC m=+567.939027411" watchObservedRunningTime="2026-07-06 15:35:46.307624627 +0000 UTC m=+567.940176609" Jul 06 15:35:46.429183 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:46.429148 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:46.527404 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:46.527370 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:48.902080 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:48.902049 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:48.997492 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:48.997456 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:57.293300 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:57.293271 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" Jul 06 15:35:57.390547 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:57.390486 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2ccg9"] Jul 06 15:35:57.390711 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:57.390683 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" podUID="d2b17cf8-07dc-4aff-93ce-158dde108d2c" containerName="limitador" containerID="cri-o://6c21266db7e3404d052acf2863f255449bf0dbec3aa03384b1b296ee4e9bdc3d" gracePeriod=30 Jul 06 15:35:57.942919 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:57.942893 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" Jul 06 15:35:57.979266 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:57.979237 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2b17cf8-07dc-4aff-93ce-158dde108d2c-config-file\") pod \"d2b17cf8-07dc-4aff-93ce-158dde108d2c\" (UID: \"d2b17cf8-07dc-4aff-93ce-158dde108d2c\") " Jul 06 15:35:57.979435 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:57.979333 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbhzg\" (UniqueName: \"kubernetes.io/projected/d2b17cf8-07dc-4aff-93ce-158dde108d2c-kube-api-access-jbhzg\") pod \"d2b17cf8-07dc-4aff-93ce-158dde108d2c\" (UID: \"d2b17cf8-07dc-4aff-93ce-158dde108d2c\") " Jul 06 15:35:57.979578 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:57.979557 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b17cf8-07dc-4aff-93ce-158dde108d2c-config-file" (OuterVolumeSpecName: "config-file") pod "d2b17cf8-07dc-4aff-93ce-158dde108d2c" (UID: "d2b17cf8-07dc-4aff-93ce-158dde108d2c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:35:57.981385 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:57.981361 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b17cf8-07dc-4aff-93ce-158dde108d2c-kube-api-access-jbhzg" (OuterVolumeSpecName: "kube-api-access-jbhzg") pod "d2b17cf8-07dc-4aff-93ce-158dde108d2c" (UID: "d2b17cf8-07dc-4aff-93ce-158dde108d2c"). InnerVolumeSpecName "kube-api-access-jbhzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:35:58.080884 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:58.080852 2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2b17cf8-07dc-4aff-93ce-158dde108d2c-config-file\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:35:58.080884 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:58.080883 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbhzg\" (UniqueName: \"kubernetes.io/projected/d2b17cf8-07dc-4aff-93ce-158dde108d2c-kube-api-access-jbhzg\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:35:58.347016 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:58.346934 2561 generic.go:356] "Generic (PLEG): container finished" podID="d2b17cf8-07dc-4aff-93ce-158dde108d2c" containerID="6c21266db7e3404d052acf2863f255449bf0dbec3aa03384b1b296ee4e9bdc3d" exitCode=0 Jul 06 15:35:58.347016 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:58.346973 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" Jul 06 15:35:58.347483 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:58.347027 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" event={"ID":"d2b17cf8-07dc-4aff-93ce-158dde108d2c","Type":"ContainerDied","Data":"6c21266db7e3404d052acf2863f255449bf0dbec3aa03384b1b296ee4e9bdc3d"} Jul 06 15:35:58.347483 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:58.347089 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-2ccg9" event={"ID":"d2b17cf8-07dc-4aff-93ce-158dde108d2c","Type":"ContainerDied","Data":"b691a50ef60f3726d3329a54d9dd0bbfe80ce1d925aeb21931f46d4cb8b8d3a5"} Jul 06 15:35:58.347483 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:58.347121 2561 scope.go:117] "RemoveContainer" containerID="6c21266db7e3404d052acf2863f255449bf0dbec3aa03384b1b296ee4e9bdc3d" Jul 06 15:35:58.356277 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:58.356261 2561 scope.go:117] "RemoveContainer" containerID="6c21266db7e3404d052acf2863f255449bf0dbec3aa03384b1b296ee4e9bdc3d" Jul 06 15:35:58.356539 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:35:58.356504 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c21266db7e3404d052acf2863f255449bf0dbec3aa03384b1b296ee4e9bdc3d\": container with ID starting with 6c21266db7e3404d052acf2863f255449bf0dbec3aa03384b1b296ee4e9bdc3d not found: ID does not exist" containerID="6c21266db7e3404d052acf2863f255449bf0dbec3aa03384b1b296ee4e9bdc3d" Jul 06 15:35:58.356592 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:58.356548 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c21266db7e3404d052acf2863f255449bf0dbec3aa03384b1b296ee4e9bdc3d"} err="failed to get container status \"6c21266db7e3404d052acf2863f255449bf0dbec3aa03384b1b296ee4e9bdc3d\": rpc error: code = NotFound desc = could not find container \"6c21266db7e3404d052acf2863f255449bf0dbec3aa03384b1b296ee4e9bdc3d\": container with ID starting with 6c21266db7e3404d052acf2863f255449bf0dbec3aa03384b1b296ee4e9bdc3d not found: ID does not exist" Jul 06 15:35:58.370766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:58.370745 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2ccg9"] Jul 06 15:35:58.373391 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:58.373367 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-2ccg9"] Jul 06 15:35:58.807444 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:35:58.807410 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2b17cf8-07dc-4aff-93ce-158dde108d2c" path="/var/lib/kubelet/pods/d2b17cf8-07dc-4aff-93ce-158dde108d2c/volumes" Jul 06 15:36:03.958865 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:03.958835 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-kmhdx"] Jul 06 15:36:03.959291 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:03.959272 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b17cf8-07dc-4aff-93ce-158dde108d2c" containerName="limitador" Jul 06 15:36:03.959366 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:03.959295 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b17cf8-07dc-4aff-93ce-158dde108d2c" containerName="limitador" Jul 06 15:36:03.959462 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:03.959448 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b17cf8-07dc-4aff-93ce-158dde108d2c" containerName="limitador" Jul 06 15:36:03.963447 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:03.963431 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-kmhdx" Jul 06 15:36:03.966150 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:03.966131 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7jqgr\"" Jul 06 15:36:03.966285 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:03.966174 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 06 15:36:03.970395 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:03.970372 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-kmhdx"] Jul 06 15:36:03.992264 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:03.992238 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:04.029034 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:04.029011 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhf2s\" (UniqueName: \"kubernetes.io/projected/19f75d9b-f72a-4dea-ab21-87a689803e9b-kube-api-access-zhf2s\") pod \"postgres-69fb4594cf-kmhdx\" (UID: \"19f75d9b-f72a-4dea-ab21-87a689803e9b\") " pod="opendatahub/postgres-69fb4594cf-kmhdx" Jul 06 15:36:04.029119 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:04.029050 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/19f75d9b-f72a-4dea-ab21-87a689803e9b-data\") pod \"postgres-69fb4594cf-kmhdx\" (UID: \"19f75d9b-f72a-4dea-ab21-87a689803e9b\") " pod="opendatahub/postgres-69fb4594cf-kmhdx" Jul 06 15:36:04.130329 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:04.130298 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhf2s\" (UniqueName: \"kubernetes.io/projected/19f75d9b-f72a-4dea-ab21-87a689803e9b-kube-api-access-zhf2s\") pod \"postgres-69fb4594cf-kmhdx\" (UID: \"19f75d9b-f72a-4dea-ab21-87a689803e9b\") " pod="opendatahub/postgres-69fb4594cf-kmhdx" Jul 06 15:36:04.130469 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:04.130350 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/19f75d9b-f72a-4dea-ab21-87a689803e9b-data\") pod \"postgres-69fb4594cf-kmhdx\" (UID: \"19f75d9b-f72a-4dea-ab21-87a689803e9b\") " pod="opendatahub/postgres-69fb4594cf-kmhdx" Jul 06 15:36:04.130670 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:04.130653 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/19f75d9b-f72a-4dea-ab21-87a689803e9b-data\") pod \"postgres-69fb4594cf-kmhdx\" (UID: \"19f75d9b-f72a-4dea-ab21-87a689803e9b\") " pod="opendatahub/postgres-69fb4594cf-kmhdx" Jul 06 15:36:04.139364 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:04.139342 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhf2s\" (UniqueName: \"kubernetes.io/projected/19f75d9b-f72a-4dea-ab21-87a689803e9b-kube-api-access-zhf2s\") pod \"postgres-69fb4594cf-kmhdx\" (UID: \"19f75d9b-f72a-4dea-ab21-87a689803e9b\") " pod="opendatahub/postgres-69fb4594cf-kmhdx" Jul 06 15:36:04.275712 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:04.275684 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-kmhdx" Jul 06 15:36:04.413068 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:04.410763 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-kmhdx"] Jul 06 15:36:04.416027 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:36:04.416001 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f75d9b_f72a_4dea_ab21_87a689803e9b.slice/crio-40d19b180cd5962f9d1fafd6d3ce3ddca9c7effb50e7b0ad65d430265af6f074 WatchSource:0}: Error finding container 40d19b180cd5962f9d1fafd6d3ce3ddca9c7effb50e7b0ad65d430265af6f074: Status 404 returned error can't find the container with id 40d19b180cd5962f9d1fafd6d3ce3ddca9c7effb50e7b0ad65d430265af6f074 Jul 06 15:36:05.385341 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:05.385295 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-kmhdx" event={"ID":"19f75d9b-f72a-4dea-ab21-87a689803e9b","Type":"ContainerStarted","Data":"40d19b180cd5962f9d1fafd6d3ce3ddca9c7effb50e7b0ad65d430265af6f074"} Jul 06 15:36:10.416937 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:10.416903 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-kmhdx" event={"ID":"19f75d9b-f72a-4dea-ab21-87a689803e9b","Type":"ContainerStarted","Data":"a482f2f1910ff8d2dc964091aaf6710f864a733f6dc7ca7d3b5fc98d1b6cd749"} Jul 06 15:36:10.417310 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:10.417026 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-kmhdx" Jul 06 15:36:10.438428 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:10.438373 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-kmhdx" podStartSLOduration=2.428736487 podStartE2EDuration="7.438352676s" totalImagesPullingTime="5.009616189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:36:10.434203449 +0000 UTC m=+592.066755435" watchObservedRunningTime="2026-07-06 15:36:10.438352676 +0000 UTC m=+592.070904661" Jul 06 15:36:16.452140 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:16.452108 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-kmhdx" Jul 06 15:36:16.778310 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:16.778276 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:16.912124 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:16.912092 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:18.735486 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:18.735457 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 15:36:18.738597 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:18.738568 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 15:36:23.759194 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:23.759130 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:23.843603 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:23.843573 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:23.917605 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:23.917574 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:23.943942 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:23.943916 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-957b6d485-g75v5"] Jul 06 15:36:23.947700 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:23.947679 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-957b6d485-g75v5" Jul 06 15:36:23.950831 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:23.950782 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 15:36:23.950831 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:23.950822 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 15:36:23.951004 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:23.950875 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tbd4d\"" Jul 06 15:36:23.956978 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:23.956936 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-957b6d485-g75v5"] Jul 06 15:36:24.012936 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:24.012866 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrqdr\" (UniqueName: \"kubernetes.io/projected/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588-kube-api-access-vrqdr\") pod \"maas-controller-957b6d485-g75v5\" (UID: \"e7d5075e-8b3f-48b1-8a47-96bfcc3c5588\") " pod="opendatahub/maas-controller-957b6d485-g75v5" Jul 06 15:36:24.013069 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:24.012992 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588-webhook-cert\") pod \"maas-controller-957b6d485-g75v5\" (UID: \"e7d5075e-8b3f-48b1-8a47-96bfcc3c5588\") " pod="opendatahub/maas-controller-957b6d485-g75v5" Jul 06 15:36:24.114214 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:24.114190 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588-webhook-cert\") pod \"maas-controller-957b6d485-g75v5\" (UID: \"e7d5075e-8b3f-48b1-8a47-96bfcc3c5588\") " pod="opendatahub/maas-controller-957b6d485-g75v5" Jul 06 15:36:24.114339 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:24.114236 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrqdr\" (UniqueName: \"kubernetes.io/projected/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588-kube-api-access-vrqdr\") pod \"maas-controller-957b6d485-g75v5\" (UID: \"e7d5075e-8b3f-48b1-8a47-96bfcc3c5588\") " pod="opendatahub/maas-controller-957b6d485-g75v5" Jul 06 15:36:24.114388 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:36:24.114337 2561 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 06 15:36:24.114425 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:36:24.114398 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588-webhook-cert podName:e7d5075e-8b3f-48b1-8a47-96bfcc3c5588 nodeName:}" failed. No retries permitted until 2026-07-06 15:36:24.614380856 +0000 UTC m=+606.246932817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588-webhook-cert") pod "maas-controller-957b6d485-g75v5" (UID: "e7d5075e-8b3f-48b1-8a47-96bfcc3c5588") : secret "maas-controller-webhook-cert" not found Jul 06 15:36:24.127552 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:24.127499 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrqdr\" (UniqueName: \"kubernetes.io/projected/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588-kube-api-access-vrqdr\") pod \"maas-controller-957b6d485-g75v5\" (UID: \"e7d5075e-8b3f-48b1-8a47-96bfcc3c5588\") " pod="opendatahub/maas-controller-957b6d485-g75v5" Jul 06 15:36:24.340235 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:24.340142 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:24.619836 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:24.619761 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588-webhook-cert\") pod \"maas-controller-957b6d485-g75v5\" (UID: \"e7d5075e-8b3f-48b1-8a47-96bfcc3c5588\") " pod="opendatahub/maas-controller-957b6d485-g75v5" Jul 06 15:36:24.622267 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:24.622244 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588-webhook-cert\") pod \"maas-controller-957b6d485-g75v5\" (UID: \"e7d5075e-8b3f-48b1-8a47-96bfcc3c5588\") " pod="opendatahub/maas-controller-957b6d485-g75v5" Jul 06 15:36:24.859496 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:24.859464 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-957b6d485-g75v5" Jul 06 15:36:24.994102 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:24.994077 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-957b6d485-g75v5"] Jul 06 15:36:24.995700 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:36:24.995662 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d5075e_8b3f_48b1_8a47_96bfcc3c5588.slice/crio-84897c540977bd56e2657072de40b9f5bfc01988417abea63e8c11353b9df239 WatchSource:0}: Error finding container 84897c540977bd56e2657072de40b9f5bfc01988417abea63e8c11353b9df239: Status 404 returned error can't find the container with id 84897c540977bd56e2657072de40b9f5bfc01988417abea63e8c11353b9df239 Jul 06 15:36:25.488586 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:25.488552 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-957b6d485-g75v5" event={"ID":"e7d5075e-8b3f-48b1-8a47-96bfcc3c5588","Type":"ContainerStarted","Data":"84897c540977bd56e2657072de40b9f5bfc01988417abea63e8c11353b9df239"} Jul 06 15:36:27.497497 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:27.497461 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:27.501235 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:27.501206 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-957b6d485-g75v5" event={"ID":"e7d5075e-8b3f-48b1-8a47-96bfcc3c5588","Type":"ContainerStarted","Data":"5a38c77caa375a4989d8d425b441d6e0f9db8b1dadc740db1ea7c0a602d4fca5"} Jul 06 15:36:27.501353 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:27.501265 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-957b6d485-g75v5" Jul 06 15:36:27.524115 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:27.524067 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-957b6d485-g75v5" podStartSLOduration=2.129966267 podStartE2EDuration="4.524048948s" totalImagesPullingTime="2.394082681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:36:27.521210261 +0000 UTC m=+609.153762244" watchObservedRunningTime="2026-07-06 15:36:27.524048948 +0000 UTC m=+609.156600935" Jul 06 15:36:27.595447 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:27.595410 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:31.937545 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:31.937496 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:32.002145 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:32.002115 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:38.513306 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:38.513272 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-957b6d485-g75v5" Jul 06 15:36:51.670730 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:51.670643 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:51.734317 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:51.734281 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:59.193587 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.193545 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:59.644501 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.644469 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:59.693889 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.693847 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-68d58c4db6-p9kws"] Jul 06 15:36:59.699611 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.699589 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-68d58c4db6-p9kws" Jul 06 15:36:59.703988 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.703967 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 06 15:36:59.714837 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.714816 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-68d58c4db6-p9kws"] Jul 06 15:36:59.836965 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.836936 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0-maas-api-tls\") pod \"maas-api-68d58c4db6-p9kws\" (UID: \"3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0\") " pod="opendatahub/maas-api-68d58c4db6-p9kws" Jul 06 15:36:59.837158 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.837061 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0-openshift-service-ca\") pod \"maas-api-68d58c4db6-p9kws\" (UID: \"3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0\") " pod="opendatahub/maas-api-68d58c4db6-p9kws" Jul 06 15:36:59.837158 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.837088 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59v8f\" (UniqueName: \"kubernetes.io/projected/3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0-kube-api-access-59v8f\") pod \"maas-api-68d58c4db6-p9kws\" (UID: \"3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0\") " pod="opendatahub/maas-api-68d58c4db6-p9kws" Jul 06 15:36:59.874617 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.874585 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:59.938038 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.937961 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0-openshift-service-ca\") pod \"maas-api-68d58c4db6-p9kws\" (UID: \"3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0\") " pod="opendatahub/maas-api-68d58c4db6-p9kws" Jul 06 15:36:59.938260 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.938234 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59v8f\" (UniqueName: \"kubernetes.io/projected/3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0-kube-api-access-59v8f\") pod \"maas-api-68d58c4db6-p9kws\" (UID: \"3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0\") " pod="opendatahub/maas-api-68d58c4db6-p9kws" Jul 06 15:36:59.938370 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.938355 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0-maas-api-tls\") pod \"maas-api-68d58c4db6-p9kws\" (UID: \"3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0\") " pod="opendatahub/maas-api-68d58c4db6-p9kws" Jul 06 15:36:59.938679 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.938653 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0-openshift-service-ca\") pod \"maas-api-68d58c4db6-p9kws\" (UID: \"3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0\") " pod="opendatahub/maas-api-68d58c4db6-p9kws" Jul 06 15:36:59.940841 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.940821 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0-maas-api-tls\") pod \"maas-api-68d58c4db6-p9kws\" (UID: \"3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0\") " pod="opendatahub/maas-api-68d58c4db6-p9kws" Jul 06 15:36:59.952917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:36:59.952888 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59v8f\" (UniqueName: \"kubernetes.io/projected/3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0-kube-api-access-59v8f\") pod \"maas-api-68d58c4db6-p9kws\" (UID: \"3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0\") " pod="opendatahub/maas-api-68d58c4db6-p9kws" Jul 06 15:37:00.011742 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.011717 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-68d58c4db6-p9kws" Jul 06 15:37:00.170727 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.170705 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-68d58c4db6-p9kws"] Jul 06 15:37:00.172177 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:37:00.172143 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3879af8c_ce8b_4b3d_a66d_490cb2bfa2b0.slice/crio-06e431553200ef00081f930e96f590bf142df6c280dea1a51e461ed3172ebcc7 WatchSource:0}: Error finding container 06e431553200ef00081f930e96f590bf142df6c280dea1a51e461ed3172ebcc7: Status 404 returned error can't find the container with id 06e431553200ef00081f930e96f590bf142df6c280dea1a51e461ed3172ebcc7 Jul 06 15:37:00.270555 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.270487 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:00.371906 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.371871 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:00.379051 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.379030 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-lr7lt"] Jul 06 15:37:00.382766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.382749 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-lr7lt" Jul 06 15:37:00.385613 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.385595 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 15:37:00.393888 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.393864 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-lr7lt"] Jul 06 15:37:00.443714 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.443683 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8d7\" (UniqueName: \"kubernetes.io/projected/82d1b0f6-6ce2-46bf-acc3-61d410363776-kube-api-access-7v8d7\") pod \"payload-processing-58f86c7798-lr7lt\" (UID: \"82d1b0f6-6ce2-46bf-acc3-61d410363776\") " pod="openshift-ingress/payload-processing-58f86c7798-lr7lt" Jul 06 15:37:00.443862 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.443838 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d1b0f6-6ce2-46bf-acc3-61d410363776-plugins-config-volume\") pod \"payload-processing-58f86c7798-lr7lt\" (UID: \"82d1b0f6-6ce2-46bf-acc3-61d410363776\") " pod="openshift-ingress/payload-processing-58f86c7798-lr7lt" Jul 06 15:37:00.544917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.544829 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d1b0f6-6ce2-46bf-acc3-61d410363776-plugins-config-volume\") pod \"payload-processing-58f86c7798-lr7lt\" (UID: \"82d1b0f6-6ce2-46bf-acc3-61d410363776\") " pod="openshift-ingress/payload-processing-58f86c7798-lr7lt" Jul 06 15:37:00.544917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.544893 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8d7\" (UniqueName: \"kubernetes.io/projected/82d1b0f6-6ce2-46bf-acc3-61d410363776-kube-api-access-7v8d7\") pod \"payload-processing-58f86c7798-lr7lt\" (UID: \"82d1b0f6-6ce2-46bf-acc3-61d410363776\") " pod="openshift-ingress/payload-processing-58f86c7798-lr7lt" Jul 06 15:37:00.545572 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.545551 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d1b0f6-6ce2-46bf-acc3-61d410363776-plugins-config-volume\") pod \"payload-processing-58f86c7798-lr7lt\" (UID: \"82d1b0f6-6ce2-46bf-acc3-61d410363776\") " pod="openshift-ingress/payload-processing-58f86c7798-lr7lt" Jul 06 15:37:00.555998 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.555976 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8d7\" (UniqueName: \"kubernetes.io/projected/82d1b0f6-6ce2-46bf-acc3-61d410363776-kube-api-access-7v8d7\") pod \"payload-processing-58f86c7798-lr7lt\" (UID: \"82d1b0f6-6ce2-46bf-acc3-61d410363776\") " pod="openshift-ingress/payload-processing-58f86c7798-lr7lt" Jul 06 15:37:00.660856 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.660825 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-68d58c4db6-p9kws" event={"ID":"3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0","Type":"ContainerStarted","Data":"06e431553200ef00081f930e96f590bf142df6c280dea1a51e461ed3172ebcc7"} Jul 06 15:37:00.693738 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.693701 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-lr7lt" Jul 06 15:37:00.846179 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:00.846148 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-lr7lt"] Jul 06 15:37:00.848128 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:37:00.848096 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d1b0f6_6ce2_46bf_acc3_61d410363776.slice/crio-8ab602f1ea67ddee3c6f0af8cecab8c91c4b562df8c9c509e82486e920deddbf WatchSource:0}: Error finding container 8ab602f1ea67ddee3c6f0af8cecab8c91c4b562df8c9c509e82486e920deddbf: Status 404 returned error can't find the container with id 8ab602f1ea67ddee3c6f0af8cecab8c91c4b562df8c9c509e82486e920deddbf Jul 06 15:37:01.668806 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:01.668755 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-lr7lt" event={"ID":"82d1b0f6-6ce2-46bf-acc3-61d410363776","Type":"ContainerStarted","Data":"8ab602f1ea67ddee3c6f0af8cecab8c91c4b562df8c9c509e82486e920deddbf"} Jul 06 15:37:02.675595 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:02.675560 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-68d58c4db6-p9kws" event={"ID":"3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0","Type":"ContainerStarted","Data":"e8efa042919be886660c74f3ffc84c37fefe78c6fdfa8ca3bf56ec3d4d1e7d0f"} Jul 06 15:37:02.676023 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:02.675665 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-68d58c4db6-p9kws" Jul 06 15:37:02.696622 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:02.696572 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-68d58c4db6-p9kws" podStartSLOduration=2.00672959 podStartE2EDuration="3.69655698s" totalImagesPullingTime="1.68982739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:37:02.695582141 +0000 UTC m=+644.328134126" watchObservedRunningTime="2026-07-06 15:37:02.69655698 +0000 UTC m=+644.329108954" Jul 06 15:37:03.681991 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:03.681953 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-lr7lt" event={"ID":"82d1b0f6-6ce2-46bf-acc3-61d410363776","Type":"ContainerStarted","Data":"1f232fe555b6673a98a68a3762653fdcc8fa15da340f7d38cc751e6c23a8011c"} Jul 06 15:37:03.682431 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:03.682207 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-lr7lt" Jul 06 15:37:03.702833 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:03.702793 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-lr7lt" podStartSLOduration=1.719851438 podStartE2EDuration="3.702778451s" totalImagesPullingTime="1.982927013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:37:03.701398733 +0000 UTC m=+645.333950717" watchObservedRunningTime="2026-07-06 15:37:03.702778451 +0000 UTC m=+645.335330433" Jul 06 15:37:08.687371 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:08.687347 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-68d58c4db6-p9kws" Jul 06 15:37:08.709003 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:08.708957 2561 generic.go:356] "Generic (PLEG): container finished" podID="d9acfb2a-5f27-4ead-8402-a7a9d253d5ed" containerID="1701f518e84b4b3c25462cd26338b7318b3df658c88df31275078f3dcb60e92a" exitCode=0 Jul 06 15:37:08.709150 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:08.709008 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" event={"ID":"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed","Type":"ContainerDied","Data":"1701f518e84b4b3c25462cd26338b7318b3df658c88df31275078f3dcb60e92a"} Jul 06 15:37:08.709387 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:08.709371 2561 scope.go:117] "RemoveContainer" containerID="1701f518e84b4b3c25462cd26338b7318b3df658c88df31275078f3dcb60e92a" Jul 06 15:37:08.710637 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:08.710603 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:37:08.710748 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:08.710684 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:37:08.710748 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:08.710732 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:37:09.718835 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:09.718788 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" event={"ID":"d9acfb2a-5f27-4ead-8402-a7a9d253d5ed","Type":"ContainerStarted","Data":"010db6e1ba538bd4f7718d2837588d9edc017d534ae31fd4125076bc8fe55488"} Jul 06 15:37:10.155169 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:10.155131 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:37:10.156365 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:10.156337 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-5nsgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 06 15:37:10.156507 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:10.156403 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" podUID="d9acfb2a-5f27-4ead-8402-a7a9d253d5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 06 15:37:10.237287 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:10.237248 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:10.338543 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:10.338500 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:11.154788 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:11.154759 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-5nsgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 06 15:37:11.155146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:11.154810 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" podUID="d9acfb2a-5f27-4ead-8402-a7a9d253d5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 06 15:37:12.154513 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:12.154469 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-5nsgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 06 15:37:12.154712 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:12.154568 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" podUID="d9acfb2a-5f27-4ead-8402-a7a9d253d5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 06 15:37:12.264215 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:12.264183 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:37:13.154757 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:13.154726 2561 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-5nsgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 06 15:37:13.154937 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:13.154786 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" podUID="d9acfb2a-5f27-4ead-8402-a7a9d253d5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 06 15:37:14.159148 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:14.159119 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:37:14.160279 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:14.160259 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-5nsgf" Jul 06 15:37:14.688567 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:14.688514 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-lr7lt" Jul 06 15:37:26.353514 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:26.353482 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:36.670782 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.670742 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh"] Jul 06 15:37:36.677593 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.677571 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.680582 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.680564 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 15:37:36.680721 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.680670 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 15:37:36.682013 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.681987 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 15:37:36.682092 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.682010 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f47hh\"" Jul 06 15:37:36.688034 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.688013 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh"] Jul 06 15:37:36.720725 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.720685 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:36.776413 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.776386 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.776413 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.776416 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfjvx\" (UniqueName: \"kubernetes.io/projected/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-kube-api-access-xfjvx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.776609 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.776444 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.776609 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.776552 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.776609 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.776604 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.776711 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.776682 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.877474 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.877447 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.877648 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.877503 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.877648 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.877557 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfjvx\" (UniqueName: \"kubernetes.io/projected/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-kube-api-access-xfjvx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.877648 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.877598 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.877648 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.877642 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.877890 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.877696 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.877974 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.877927 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.878078 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.878058 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.878137 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.878067 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.880019 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.879996 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.880279 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.880262 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.887717 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.887696 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfjvx\" (UniqueName: \"kubernetes.io/projected/fe9f4afd-8eff-411d-a9cb-8c93ee3ec521-kube-api-access-xfjvx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jqhkh\" (UID: \"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:36.989251 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:36.989224 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:37.142921 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:37.142889 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh"] Jul 06 15:37:37.143884 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:37:37.143850 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9f4afd_8eff_411d_a9cb_8c93ee3ec521.slice/crio-dafe509e50ba7862b2e3fb4c77c4a808ec2b48696876093eecd963f3503fc2a4 WatchSource:0}: Error finding container dafe509e50ba7862b2e3fb4c77c4a808ec2b48696876093eecd963f3503fc2a4: Status 404 returned error can't find the container with id dafe509e50ba7862b2e3fb4c77c4a808ec2b48696876093eecd963f3503fc2a4 Jul 06 15:37:37.866274 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:37.866235 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" event={"ID":"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521","Type":"ContainerStarted","Data":"dafe509e50ba7862b2e3fb4c77c4a808ec2b48696876093eecd963f3503fc2a4"} Jul 06 15:37:38.873303 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:38.873266 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" event={"ID":"fe9f4afd-8eff-411d-a9cb-8c93ee3ec521","Type":"ContainerStarted","Data":"a83fcacc3ef9bb6854c0a99272ebaecd893cb8cb407bda08d30313f868d8f50c"} Jul 06 15:37:38.897429 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:38.897377 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" podStartSLOduration=1.820011899 podStartE2EDuration="2.897361391s" totalImagesPullingTime="1.077349492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:37:38.895924609 +0000 UTC m=+680.528476593" watchObservedRunningTime="2026-07-06 15:37:38.897361391 +0000 UTC m=+680.529913375" Jul 06 15:37:44.892214 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:44.892176 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:46.990368 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:46.990335 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:46.990854 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:46.990391 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:47.002875 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:47.002852 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:47.929637 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:47.929610 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jqhkh" Jul 06 15:37:55.549382 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:37:55.549347 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:38:05.409965 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:38:05.409937 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:38:12.217138 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:38:12.217062 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:38:27.092926 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:38:27.092879 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:38:27.166745 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:38:27.166712 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:38:58.022560 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:38:58.022483 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:38:58.092201 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:38:58.092167 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:39:01.736514 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:39:01.736477 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:39:03.113068 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:39:03.113024 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:39:03.213970 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:39:03.213933 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:39:04.268224 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:39:04.268182 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:39:04.353330 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:39:04.353286 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:03.980308 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:03.980275 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:17.485822 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:17.485786 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:17.578032 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:17.577990 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-957b6d485-g75v5"] Jul 06 15:40:17.578240 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:17.578214 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-957b6d485-g75v5" podUID="e7d5075e-8b3f-48b1-8a47-96bfcc3c5588" containerName="manager" containerID="cri-o://5a38c77caa375a4989d8d425b441d6e0f9db8b1dadc740db1ea7c0a602d4fca5" gracePeriod=30 Jul 06 15:40:17.828958 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:17.828934 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-957b6d485-g75v5" Jul 06 15:40:17.934424 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:17.934380 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrqdr\" (UniqueName: \"kubernetes.io/projected/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588-kube-api-access-vrqdr\") pod \"e7d5075e-8b3f-48b1-8a47-96bfcc3c5588\" (UID: \"e7d5075e-8b3f-48b1-8a47-96bfcc3c5588\") " Jul 06 15:40:17.934648 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:17.934507 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588-webhook-cert\") pod \"e7d5075e-8b3f-48b1-8a47-96bfcc3c5588\" (UID: \"e7d5075e-8b3f-48b1-8a47-96bfcc3c5588\") " Jul 06 15:40:17.936769 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:17.936734 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588-kube-api-access-vrqdr" (OuterVolumeSpecName: "kube-api-access-vrqdr") pod "e7d5075e-8b3f-48b1-8a47-96bfcc3c5588" (UID: "e7d5075e-8b3f-48b1-8a47-96bfcc3c5588"). InnerVolumeSpecName "kube-api-access-vrqdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:40:17.936877 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:17.936825 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e7d5075e-8b3f-48b1-8a47-96bfcc3c5588" (UID: "e7d5075e-8b3f-48b1-8a47-96bfcc3c5588"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:40:18.036020 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:18.035959 2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588-webhook-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:40:18.036020 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:18.035981 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrqdr\" (UniqueName: \"kubernetes.io/projected/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588-kube-api-access-vrqdr\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:40:18.680876 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:18.680850 2561 generic.go:356] "Generic (PLEG): container finished" podID="e7d5075e-8b3f-48b1-8a47-96bfcc3c5588" containerID="5a38c77caa375a4989d8d425b441d6e0f9db8b1dadc740db1ea7c0a602d4fca5" exitCode=0 Jul 06 15:40:18.681266 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:18.680896 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-957b6d485-g75v5" Jul 06 15:40:18.681266 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:18.680935 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-957b6d485-g75v5" event={"ID":"e7d5075e-8b3f-48b1-8a47-96bfcc3c5588","Type":"ContainerDied","Data":"5a38c77caa375a4989d8d425b441d6e0f9db8b1dadc740db1ea7c0a602d4fca5"} Jul 06 15:40:18.681266 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:18.680985 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-957b6d485-g75v5" event={"ID":"e7d5075e-8b3f-48b1-8a47-96bfcc3c5588","Type":"ContainerDied","Data":"84897c540977bd56e2657072de40b9f5bfc01988417abea63e8c11353b9df239"} Jul 06 15:40:18.681266 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:18.681011 2561 scope.go:117] "RemoveContainer" containerID="5a38c77caa375a4989d8d425b441d6e0f9db8b1dadc740db1ea7c0a602d4fca5" Jul 06 15:40:18.690829 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:18.690811 2561 scope.go:117] "RemoveContainer" containerID="5a38c77caa375a4989d8d425b441d6e0f9db8b1dadc740db1ea7c0a602d4fca5" Jul 06 15:40:18.691104 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:40:18.691083 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a38c77caa375a4989d8d425b441d6e0f9db8b1dadc740db1ea7c0a602d4fca5\": container with ID starting with 5a38c77caa375a4989d8d425b441d6e0f9db8b1dadc740db1ea7c0a602d4fca5 not found: ID does not exist" containerID="5a38c77caa375a4989d8d425b441d6e0f9db8b1dadc740db1ea7c0a602d4fca5" Jul 06 15:40:18.691185 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:18.691116 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a38c77caa375a4989d8d425b441d6e0f9db8b1dadc740db1ea7c0a602d4fca5"} err="failed to get container status \"5a38c77caa375a4989d8d425b441d6e0f9db8b1dadc740db1ea7c0a602d4fca5\": rpc error: code = NotFound desc = could not find container \"5a38c77caa375a4989d8d425b441d6e0f9db8b1dadc740db1ea7c0a602d4fca5\": container with ID starting with 5a38c77caa375a4989d8d425b441d6e0f9db8b1dadc740db1ea7c0a602d4fca5 not found: ID does not exist" Jul 06 15:40:18.712448 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:18.712427 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-957b6d485-g75v5"] Jul 06 15:40:18.721364 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:18.721338 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-957b6d485-g75v5"] Jul 06 15:40:18.805278 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:18.805250 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7d5075e-8b3f-48b1-8a47-96bfcc3c5588" path="/var/lib/kubelet/pods/e7d5075e-8b3f-48b1-8a47-96bfcc3c5588/volumes" Jul 06 15:40:19.423309 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.423277 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:19.532972 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.532944 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-957b6d485-2jkhc"] Jul 06 15:40:19.533350 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.533336 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d5075e-8b3f-48b1-8a47-96bfcc3c5588" containerName="manager" Jul 06 15:40:19.533403 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.533352 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d5075e-8b3f-48b1-8a47-96bfcc3c5588" containerName="manager" Jul 06 15:40:19.533451 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.533429 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d5075e-8b3f-48b1-8a47-96bfcc3c5588" containerName="manager" Jul 06 15:40:19.537792 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.537773 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-957b6d485-2jkhc" Jul 06 15:40:19.541970 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.541937 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 15:40:19.541970 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.541946 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tbd4d\"" Jul 06 15:40:19.542124 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.541948 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 15:40:19.546734 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.546712 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-957b6d485-2jkhc"] Jul 06 15:40:19.649779 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.649748 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mkps\" (UniqueName: \"kubernetes.io/projected/ecf4090d-7bf0-4bff-a45b-9f91518ce855-kube-api-access-8mkps\") pod \"maas-controller-957b6d485-2jkhc\" (UID: \"ecf4090d-7bf0-4bff-a45b-9f91518ce855\") " pod="opendatahub/maas-controller-957b6d485-2jkhc" Jul 06 15:40:19.649927 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.649801 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecf4090d-7bf0-4bff-a45b-9f91518ce855-webhook-cert\") pod \"maas-controller-957b6d485-2jkhc\" (UID: \"ecf4090d-7bf0-4bff-a45b-9f91518ce855\") " pod="opendatahub/maas-controller-957b6d485-2jkhc" Jul 06 15:40:19.751019 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.750995 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkps\" (UniqueName: \"kubernetes.io/projected/ecf4090d-7bf0-4bff-a45b-9f91518ce855-kube-api-access-8mkps\") pod \"maas-controller-957b6d485-2jkhc\" (UID: \"ecf4090d-7bf0-4bff-a45b-9f91518ce855\") " pod="opendatahub/maas-controller-957b6d485-2jkhc" Jul 06 15:40:19.751450 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.751039 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecf4090d-7bf0-4bff-a45b-9f91518ce855-webhook-cert\") pod \"maas-controller-957b6d485-2jkhc\" (UID: \"ecf4090d-7bf0-4bff-a45b-9f91518ce855\") " pod="opendatahub/maas-controller-957b6d485-2jkhc" Jul 06 15:40:19.753485 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.753467 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecf4090d-7bf0-4bff-a45b-9f91518ce855-webhook-cert\") pod \"maas-controller-957b6d485-2jkhc\" (UID: \"ecf4090d-7bf0-4bff-a45b-9f91518ce855\") " pod="opendatahub/maas-controller-957b6d485-2jkhc" Jul 06 15:40:19.762224 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.762204 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkps\" (UniqueName: \"kubernetes.io/projected/ecf4090d-7bf0-4bff-a45b-9f91518ce855-kube-api-access-8mkps\") pod \"maas-controller-957b6d485-2jkhc\" (UID: \"ecf4090d-7bf0-4bff-a45b-9f91518ce855\") " pod="opendatahub/maas-controller-957b6d485-2jkhc" Jul 06 15:40:19.849014 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.848988 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-957b6d485-2jkhc" Jul 06 15:40:19.988954 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.988933 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-957b6d485-2jkhc"] Jul 06 15:40:19.990920 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:40:19.990891 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf4090d_7bf0_4bff_a45b_9f91518ce855.slice/crio-7de9450e0c265ca52231a106e3d5a0831f8b81c8edc3e198904ae5436d75ba70 WatchSource:0}: Error finding container 7de9450e0c265ca52231a106e3d5a0831f8b81c8edc3e198904ae5436d75ba70: Status 404 returned error can't find the container with id 7de9450e0c265ca52231a106e3d5a0831f8b81c8edc3e198904ae5436d75ba70 Jul 06 15:40:19.992292 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:19.992274 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 15:40:20.693136 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:20.693050 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-957b6d485-2jkhc" event={"ID":"ecf4090d-7bf0-4bff-a45b-9f91518ce855","Type":"ContainerStarted","Data":"5c8beeddd585456ce0bf955f0a7e84160e425babd88a498537f0c61e4f81759d"} Jul 06 15:40:20.693136 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:20.693115 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-957b6d485-2jkhc" event={"ID":"ecf4090d-7bf0-4bff-a45b-9f91518ce855","Type":"ContainerStarted","Data":"7de9450e0c265ca52231a106e3d5a0831f8b81c8edc3e198904ae5436d75ba70"} Jul 06 15:40:20.693346 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:20.693160 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-957b6d485-2jkhc" Jul 06 15:40:20.716443 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:20.716393 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-957b6d485-2jkhc" podStartSLOduration=1.330342205 podStartE2EDuration="1.716374771s" totalImagesPullingTime="386.032566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:40:20.712587034 +0000 UTC m=+842.345139018" watchObservedRunningTime="2026-07-06 15:40:20.716374771 +0000 UTC m=+842.348926756" Jul 06 15:40:31.705920 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:40:31.705891 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-957b6d485-2jkhc" Jul 06 15:41:00.254509 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:41:00.254471 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:00.351712 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:41:00.351674 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:00.493768 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:41:00.493731 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:06.881278 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:41:06.881244 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:06.990437 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:41:06.990399 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:07.085472 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:41:07.085437 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:07.295051 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:41:07.295019 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:13.738903 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:41:13.738820 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:13.827505 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:41:13.827467 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:18.770238 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:41:18.770212 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 15:41:18.770707 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:41:18.770305 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 15:41:23.182802 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:41:23.182766 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:42:43.379860 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:42:43.379780 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:42:45.115732 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:42:45.115697 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:42:54.876173 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:42:54.876131 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:00.164196 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:00.164160 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722545-qqmh5"] Jul 06 15:45:00.167971 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:00.167948 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722545-qqmh5" Jul 06 15:45:00.171521 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:00.171501 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rszzk\"" Jul 06 15:45:00.194348 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:00.194317 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hvk7\" (UniqueName: \"kubernetes.io/projected/97ef6680-dccc-42fa-9521-346b03544ab0-kube-api-access-2hvk7\") pod \"maas-api-key-cleanup-29722545-qqmh5\" (UID: \"97ef6680-dccc-42fa-9521-346b03544ab0\") " pod="opendatahub/maas-api-key-cleanup-29722545-qqmh5" Jul 06 15:45:00.195440 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:00.195419 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722545-qqmh5"] Jul 06 15:45:00.295654 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:00.295625 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hvk7\" (UniqueName: \"kubernetes.io/projected/97ef6680-dccc-42fa-9521-346b03544ab0-kube-api-access-2hvk7\") pod \"maas-api-key-cleanup-29722545-qqmh5\" (UID: \"97ef6680-dccc-42fa-9521-346b03544ab0\") " pod="opendatahub/maas-api-key-cleanup-29722545-qqmh5" Jul 06 15:45:00.305250 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:00.305218 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hvk7\" (UniqueName: \"kubernetes.io/projected/97ef6680-dccc-42fa-9521-346b03544ab0-kube-api-access-2hvk7\") pod \"maas-api-key-cleanup-29722545-qqmh5\" (UID: \"97ef6680-dccc-42fa-9521-346b03544ab0\") " pod="opendatahub/maas-api-key-cleanup-29722545-qqmh5" Jul 06 15:45:00.478360 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:00.478314 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722545-qqmh5" Jul 06 15:45:00.825361 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:00.825339 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722545-qqmh5"] Jul 06 15:45:00.827234 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:45:00.827197 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ef6680_dccc_42fa_9521_346b03544ab0.slice/crio-9f3b785717885431d9455022b98ce7b884cf749c571b46056154fa634f1bb0d9 WatchSource:0}: Error finding container 9f3b785717885431d9455022b98ce7b884cf749c571b46056154fa634f1bb0d9: Status 404 returned error can't find the container with id 9f3b785717885431d9455022b98ce7b884cf749c571b46056154fa634f1bb0d9 Jul 06 15:45:01.097335 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:01.097247 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722545-qqmh5" event={"ID":"97ef6680-dccc-42fa-9521-346b03544ab0","Type":"ContainerStarted","Data":"9f3b785717885431d9455022b98ce7b884cf749c571b46056154fa634f1bb0d9"} Jul 06 15:45:04.116260 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:04.116228 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722545-qqmh5" event={"ID":"97ef6680-dccc-42fa-9521-346b03544ab0","Type":"ContainerStarted","Data":"295f77f20dd208705edf620963812af113de1431a6128021cc2d756c7b4e3f6e"} Jul 06 15:45:04.136796 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:04.136753 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722545-qqmh5" podStartSLOduration=1.309085882 podStartE2EDuration="4.136736439s" totalImagesPullingTime="2.827650557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:45:04.135600935 +0000 UTC m=+1125.768152919" watchObservedRunningTime="2026-07-06 15:45:04.136736439 +0000 UTC m=+1125.769288421" Jul 06 15:45:05.122799 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:05.122757 2561 generic.go:356] "Generic (PLEG): container finished" podID="97ef6680-dccc-42fa-9521-346b03544ab0" containerID="295f77f20dd208705edf620963812af113de1431a6128021cc2d756c7b4e3f6e" exitCode=0 Jul 06 15:45:05.123153 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:05.122833 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722545-qqmh5" event={"ID":"97ef6680-dccc-42fa-9521-346b03544ab0","Type":"ContainerDied","Data":"295f77f20dd208705edf620963812af113de1431a6128021cc2d756c7b4e3f6e"} Jul 06 15:45:06.262543 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:06.262505 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722545-qqmh5" Jul 06 15:45:06.352051 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:06.352024 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hvk7\" (UniqueName: \"kubernetes.io/projected/97ef6680-dccc-42fa-9521-346b03544ab0-kube-api-access-2hvk7\") pod \"97ef6680-dccc-42fa-9521-346b03544ab0\" (UID: \"97ef6680-dccc-42fa-9521-346b03544ab0\") " Jul 06 15:45:06.354201 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:06.354178 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ef6680-dccc-42fa-9521-346b03544ab0-kube-api-access-2hvk7" (OuterVolumeSpecName: "kube-api-access-2hvk7") pod "97ef6680-dccc-42fa-9521-346b03544ab0" (UID: "97ef6680-dccc-42fa-9521-346b03544ab0"). InnerVolumeSpecName "kube-api-access-2hvk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:45:06.452766 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:06.452699 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hvk7\" (UniqueName: \"kubernetes.io/projected/97ef6680-dccc-42fa-9521-346b03544ab0-kube-api-access-2hvk7\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:45:07.134206 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:07.134128 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722545-qqmh5" Jul 06 15:45:07.134206 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:07.134167 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722545-qqmh5" event={"ID":"97ef6680-dccc-42fa-9521-346b03544ab0","Type":"ContainerDied","Data":"9f3b785717885431d9455022b98ce7b884cf749c571b46056154fa634f1bb0d9"} Jul 06 15:45:07.134378 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:07.134216 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f3b785717885431d9455022b98ce7b884cf749c571b46056154fa634f1bb0d9" Jul 06 15:45:09.529578 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:09.529543 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd"] Jul 06 15:45:09.529956 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:09.529744 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" podUID="85d66866-ed02-4e93-a6d5-a6ca57a6b349" containerName="manager" containerID="cri-o://e8dea3319bac8025ce962be604dcf1ccb8ff7a5d726b34505ed41b367c1bcb89" gracePeriod=10 Jul 06 15:45:10.081716 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:10.081681 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:12.986768 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:12.986743 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" Jul 06 15:45:13.112618 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:13.112517 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tftt\" (UniqueName: \"kubernetes.io/projected/85d66866-ed02-4e93-a6d5-a6ca57a6b349-kube-api-access-7tftt\") pod \"85d66866-ed02-4e93-a6d5-a6ca57a6b349\" (UID: \"85d66866-ed02-4e93-a6d5-a6ca57a6b349\") " Jul 06 15:45:13.112835 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:13.112641 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85d66866-ed02-4e93-a6d5-a6ca57a6b349-extensions-socket-volume\") pod \"85d66866-ed02-4e93-a6d5-a6ca57a6b349\" (UID: \"85d66866-ed02-4e93-a6d5-a6ca57a6b349\") " Jul 06 15:45:13.113008 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:13.112985 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d66866-ed02-4e93-a6d5-a6ca57a6b349-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "85d66866-ed02-4e93-a6d5-a6ca57a6b349" (UID: "85d66866-ed02-4e93-a6d5-a6ca57a6b349"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:45:13.114641 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:13.114620 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d66866-ed02-4e93-a6d5-a6ca57a6b349-kube-api-access-7tftt" (OuterVolumeSpecName: "kube-api-access-7tftt") pod "85d66866-ed02-4e93-a6d5-a6ca57a6b349" (UID: "85d66866-ed02-4e93-a6d5-a6ca57a6b349"). InnerVolumeSpecName "kube-api-access-7tftt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:45:13.165855 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:13.165828 2561 generic.go:356] "Generic (PLEG): container finished" podID="85d66866-ed02-4e93-a6d5-a6ca57a6b349" containerID="e8dea3319bac8025ce962be604dcf1ccb8ff7a5d726b34505ed41b367c1bcb89" exitCode=0 Jul 06 15:45:13.165957 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:13.165871 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" Jul 06 15:45:13.165957 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:13.165901 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" event={"ID":"85d66866-ed02-4e93-a6d5-a6ca57a6b349","Type":"ContainerDied","Data":"e8dea3319bac8025ce962be604dcf1ccb8ff7a5d726b34505ed41b367c1bcb89"} Jul 06 15:45:13.165957 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:13.165940 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd" event={"ID":"85d66866-ed02-4e93-a6d5-a6ca57a6b349","Type":"ContainerDied","Data":"cdf6b6569bbd6e6e2566f41c4299a3e8604455f827141b41c8e7d1af0126ff6a"} Jul 06 15:45:13.166055 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:13.165961 2561 scope.go:117] "RemoveContainer" containerID="e8dea3319bac8025ce962be604dcf1ccb8ff7a5d726b34505ed41b367c1bcb89" Jul 06 15:45:13.178103 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:13.178079 2561 scope.go:117] "RemoveContainer" containerID="e8dea3319bac8025ce962be604dcf1ccb8ff7a5d726b34505ed41b367c1bcb89" Jul 06 15:45:13.178891 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:45:13.178863 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8dea3319bac8025ce962be604dcf1ccb8ff7a5d726b34505ed41b367c1bcb89\": container with ID starting with e8dea3319bac8025ce962be604dcf1ccb8ff7a5d726b34505ed41b367c1bcb89 not found: ID does not exist" containerID="e8dea3319bac8025ce962be604dcf1ccb8ff7a5d726b34505ed41b367c1bcb89" Jul 06 15:45:13.178987 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:13.178902 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8dea3319bac8025ce962be604dcf1ccb8ff7a5d726b34505ed41b367c1bcb89"} err="failed to get container status \"e8dea3319bac8025ce962be604dcf1ccb8ff7a5d726b34505ed41b367c1bcb89\": rpc error: code = NotFound desc = could not find container \"e8dea3319bac8025ce962be604dcf1ccb8ff7a5d726b34505ed41b367c1bcb89\": container with ID starting with e8dea3319bac8025ce962be604dcf1ccb8ff7a5d726b34505ed41b367c1bcb89 not found: ID does not exist" Jul 06 15:45:13.190115 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:13.190092 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd"] Jul 06 15:45:13.194832 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:13.194811 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fnpqd"] Jul 06 15:45:13.213585 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:13.213566 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tftt\" (UniqueName: \"kubernetes.io/projected/85d66866-ed02-4e93-a6d5-a6ca57a6b349-kube-api-access-7tftt\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:45:13.213667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:13.213590 2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85d66866-ed02-4e93-a6d5-a6ca57a6b349-extensions-socket-volume\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:45:14.799589 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:45:14.799558 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85d66866-ed02-4e93-a6d5-a6ca57a6b349" path="/var/lib/kubelet/pods/85d66866-ed02-4e93-a6d5-a6ca57a6b349/volumes" Jul 06 15:46:10.730043 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.730012 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq"] Jul 06 15:46:10.730437 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.730410 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85d66866-ed02-4e93-a6d5-a6ca57a6b349" containerName="manager" Jul 06 15:46:10.730437 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.730420 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="85d66866-ed02-4e93-a6d5-a6ca57a6b349" containerName="manager" Jul 06 15:46:10.730437 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.730433 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ef6680-dccc-42fa-9521-346b03544ab0" containerName="cleanup" Jul 06 15:46:10.730437 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.730439 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ef6680-dccc-42fa-9521-346b03544ab0" containerName="cleanup" Jul 06 15:46:10.730748 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.730521 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="85d66866-ed02-4e93-a6d5-a6ca57a6b349" containerName="manager" Jul 06 15:46:10.730748 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.730549 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ef6680-dccc-42fa-9521-346b03544ab0" containerName="cleanup" Jul 06 15:46:10.733741 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.733723 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq" Jul 06 15:46:10.736509 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.736490 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-99tx8\"" Jul 06 15:46:10.746168 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.746146 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq"] Jul 06 15:46:10.795306 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.795282 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bcbde0f-bbc4-4c5f-a089-daaeb038744d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tgcnq\" (UID: \"5bcbde0f-bbc4-4c5f-a089-daaeb038744d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq" Jul 06 15:46:10.795434 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.795335 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xpf2\" (UniqueName: \"kubernetes.io/projected/5bcbde0f-bbc4-4c5f-a089-daaeb038744d-kube-api-access-5xpf2\") pod \"kuadrant-operator-controller-manager-86878c769f-tgcnq\" (UID: \"5bcbde0f-bbc4-4c5f-a089-daaeb038744d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq" Jul 06 15:46:10.896286 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.896257 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bcbde0f-bbc4-4c5f-a089-daaeb038744d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tgcnq\" (UID: \"5bcbde0f-bbc4-4c5f-a089-daaeb038744d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq" Jul 06 15:46:10.896440 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.896323 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xpf2\" (UniqueName: \"kubernetes.io/projected/5bcbde0f-bbc4-4c5f-a089-daaeb038744d-kube-api-access-5xpf2\") pod \"kuadrant-operator-controller-manager-86878c769f-tgcnq\" (UID: \"5bcbde0f-bbc4-4c5f-a089-daaeb038744d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq" Jul 06 15:46:10.896671 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.896641 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bcbde0f-bbc4-4c5f-a089-daaeb038744d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tgcnq\" (UID: \"5bcbde0f-bbc4-4c5f-a089-daaeb038744d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq" Jul 06 15:46:10.906047 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:10.906021 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xpf2\" (UniqueName: \"kubernetes.io/projected/5bcbde0f-bbc4-4c5f-a089-daaeb038744d-kube-api-access-5xpf2\") pod \"kuadrant-operator-controller-manager-86878c769f-tgcnq\" (UID: \"5bcbde0f-bbc4-4c5f-a089-daaeb038744d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq" Jul 06 15:46:11.045357 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:11.045280 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq" Jul 06 15:46:11.183641 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:11.183619 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq"] Jul 06 15:46:11.186380 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:46:11.186346 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bcbde0f_bbc4_4c5f_a089_daaeb038744d.slice/crio-6f1c93aaebd28fd213cbfb89968cc63735e6c5b3029fda05e3e1a5dc5883a033 WatchSource:0}: Error finding container 6f1c93aaebd28fd213cbfb89968cc63735e6c5b3029fda05e3e1a5dc5883a033: Status 404 returned error can't find the container with id 6f1c93aaebd28fd213cbfb89968cc63735e6c5b3029fda05e3e1a5dc5883a033 Jul 06 15:46:11.455480 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:11.455399 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq" event={"ID":"5bcbde0f-bbc4-4c5f-a089-daaeb038744d","Type":"ContainerStarted","Data":"0b68f5372b9560e4f49269a93f03e28287bd86c8b5765d151c4b103b037fcc0b"} Jul 06 15:46:11.455480 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:11.455449 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq" event={"ID":"5bcbde0f-bbc4-4c5f-a089-daaeb038744d","Type":"ContainerStarted","Data":"6f1c93aaebd28fd213cbfb89968cc63735e6c5b3029fda05e3e1a5dc5883a033"} Jul 06 15:46:11.455704 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:11.455523 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq" Jul 06 15:46:11.489939 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:11.489898 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq" podStartSLOduration=1.4898838030000001 podStartE2EDuration="1.489883803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:46:11.487146368 +0000 UTC m=+1193.119698342" watchObservedRunningTime="2026-07-06 15:46:11.489883803 +0000 UTC m=+1193.122435786" Jul 06 15:46:18.808457 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:18.808426 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 15:46:18.809919 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:18.809897 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 15:46:22.463854 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:22.463818 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tgcnq" Jul 06 15:46:30.095395 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:46:30.095363 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:51.521438 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:49:51.521409 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:17.206648 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:50:17.206561 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:17.306677 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:50:17.306653 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:15.725955 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:51:15.725920 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:18.841076 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:51:18.841044 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 15:51:18.842900 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:51:18.842868 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 15:53:59.405571 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:53:59.405519 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:07.469575 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:07.469518 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:09.093083 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:09.093045 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:17.385946 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:17.385910 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:22.695024 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:22.694992 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:35.869270 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:35.869240 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:35.968424 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:35.968392 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:36.533833 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:36.533804 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:41.355620 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:41.355540 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:42.328756 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:42.328722 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:47.823124 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:47.823088 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:47.924923 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:47.924888 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:48.063771 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.063739 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96"] Jul 06 15:54:48.068358 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.068338 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.071724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.071706 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0ae7973b-openshift-default-dockercfg-z2qlk\"" Jul 06 15:54:48.092545 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.092503 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96"] Jul 06 15:54:48.097687 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.097666 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:48.121894 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.121870 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-token\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.122000 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.121908 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-workload-socket\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.122000 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.121928 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-credential-socket\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.122117 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.122020 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjw5z\" (UniqueName: \"kubernetes.io/projected/df2aeac3-05fa-4db0-a00e-bb2697587c9e-kube-api-access-mjw5z\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.122117 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.122103 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-workload-certs\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.122225 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.122142 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-envoy\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.122225 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.122166 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-data\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.122225 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.122195 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-podinfo\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.122367 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.122284 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istiod-ca-cert\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.188583 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.188517 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:48.223586 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.223552 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-token\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.223746 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.223609 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-workload-socket\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.223746 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.223630 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-credential-socket\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.223746 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.223657 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjw5z\" (UniqueName: \"kubernetes.io/projected/df2aeac3-05fa-4db0-a00e-bb2697587c9e-kube-api-access-mjw5z\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.223746 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.223710 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-workload-certs\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.223746 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.223746 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-envoy\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.224014 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.223770 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-data\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.224014 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.223923 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-podinfo\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.224128 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.224019 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istiod-ca-cert\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.224128 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.224068 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-workload-socket\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.224225 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.224210 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-data\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.224283 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.224226 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-credential-socket\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.224374 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.224352 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-workload-certs\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.224792 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.224766 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istiod-ca-cert\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.226182 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.226163 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-envoy\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.226565 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.226523 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-podinfo\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.233203 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.233180 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjw5z\" (UniqueName: \"kubernetes.io/projected/df2aeac3-05fa-4db0-a00e-bb2697587c9e-kube-api-access-mjw5z\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.233303 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.233230 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-token\") pod \"e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.379431 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.379361 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:48.530819 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.530789 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96"] Jul 06 15:54:48.531275 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:54:48.531254 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2aeac3_05fa_4db0_a00e_bb2697587c9e.slice/crio-4bd1ad7e39f93f5176e2a06b9df4b1bb0ff6f01b77eb4ca74471a6e1c442693b WatchSource:0}: Error finding container 4bd1ad7e39f93f5176e2a06b9df4b1bb0ff6f01b77eb4ca74471a6e1c442693b: Status 404 returned error can't find the container with id 4bd1ad7e39f93f5176e2a06b9df4b1bb0ff6f01b77eb4ca74471a6e1c442693b Jul 06 15:54:48.533264 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.533235 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:54:48.533353 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.533294 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:54:48.533353 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:48.533323 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:54:49.090504 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.090469 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" event={"ID":"df2aeac3-05fa-4db0-a00e-bb2697587c9e","Type":"ContainerStarted","Data":"1fcf1b09b950c3192280bcc5d1c49f9e06e34902a24996a817c0cf4dc1989d1c"} Jul 06 15:54:49.090902 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.090570 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" event={"ID":"df2aeac3-05fa-4db0-a00e-bb2697587c9e","Type":"ContainerStarted","Data":"4bd1ad7e39f93f5176e2a06b9df4b1bb0ff6f01b77eb4ca74471a6e1c442693b"} Jul 06 15:54:49.120570 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.120502 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" podStartSLOduration=1.120487273 podStartE2EDuration="1.120487273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:54:49.119134855 +0000 UTC m=+1710.751686837" watchObservedRunningTime="2026-07-06 15:54:49.120487273 +0000 UTC m=+1710.753039256" Jul 06 15:54:49.373329 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.373240 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:49.380243 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.380225 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:49.384968 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.384949 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:49.459067 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.459042 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:49.581278 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.581235 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6"] Jul 06 15:54:49.585022 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.584998 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" Jul 06 15:54:49.588225 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.588202 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rszzk\"" Jul 06 15:54:49.588346 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.588253 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-0ae7973b-serving-cert\"" Jul 06 15:54:49.596581 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.596559 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6"] Jul 06 15:54:49.615887 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.615865 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:49.638096 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.638024 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a11536-02d8-4c71-ac6d-b4eb052f119b-openshift-service-ca\") pod \"maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6\" (UID: \"f8a11536-02d8-4c71-ac6d-b4eb052f119b\") " pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" Jul 06 15:54:49.638096 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.638068 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a11536-02d8-4c71-ac6d-b4eb052f119b-maas-api-tls\") pod \"maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6\" (UID: \"f8a11536-02d8-4c71-ac6d-b4eb052f119b\") " pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" Jul 06 15:54:49.638246 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.638114 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s8km\" (UniqueName: \"kubernetes.io/projected/f8a11536-02d8-4c71-ac6d-b4eb052f119b-kube-api-access-5s8km\") pod \"maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6\" (UID: \"f8a11536-02d8-4c71-ac6d-b4eb052f119b\") " pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" Jul 06 15:54:49.739274 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.739247 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s8km\" (UniqueName: \"kubernetes.io/projected/f8a11536-02d8-4c71-ac6d-b4eb052f119b-kube-api-access-5s8km\") pod \"maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6\" (UID: \"f8a11536-02d8-4c71-ac6d-b4eb052f119b\") " pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" Jul 06 15:54:49.739430 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.739360 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a11536-02d8-4c71-ac6d-b4eb052f119b-openshift-service-ca\") pod \"maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6\" (UID: \"f8a11536-02d8-4c71-ac6d-b4eb052f119b\") " pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" Jul 06 15:54:49.739430 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.739409 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a11536-02d8-4c71-ac6d-b4eb052f119b-maas-api-tls\") pod \"maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6\" (UID: \"f8a11536-02d8-4c71-ac6d-b4eb052f119b\") " pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" Jul 06 15:54:49.740038 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.740012 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a11536-02d8-4c71-ac6d-b4eb052f119b-openshift-service-ca\") pod \"maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6\" (UID: \"f8a11536-02d8-4c71-ac6d-b4eb052f119b\") " pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" Jul 06 15:54:49.742067 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.742047 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a11536-02d8-4c71-ac6d-b4eb052f119b-maas-api-tls\") pod \"maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6\" (UID: \"f8a11536-02d8-4c71-ac6d-b4eb052f119b\") " pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" Jul 06 15:54:49.750145 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.750126 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s8km\" (UniqueName: \"kubernetes.io/projected/f8a11536-02d8-4c71-ac6d-b4eb052f119b-kube-api-access-5s8km\") pod \"maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6\" (UID: \"f8a11536-02d8-4c71-ac6d-b4eb052f119b\") " pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" Jul 06 15:54:49.784137 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.784116 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:49.897983 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:49.897916 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" Jul 06 15:54:50.038931 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:50.038905 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6"] Jul 06 15:54:50.039940 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:54:50.039913 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a11536_02d8_4c71_ac6d_b4eb052f119b.slice/crio-8f28786efbbcc16aaa5fa58a623970f11be01d7388da126cf784fcccc05f94e3 WatchSource:0}: Error finding container 8f28786efbbcc16aaa5fa58a623970f11be01d7388da126cf784fcccc05f94e3: Status 404 returned error can't find the container with id 8f28786efbbcc16aaa5fa58a623970f11be01d7388da126cf784fcccc05f94e3 Jul 06 15:54:50.041481 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:50.041459 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 15:54:50.097944 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:50.097916 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" event={"ID":"f8a11536-02d8-4c71-ac6d-b4eb052f119b","Type":"ContainerStarted","Data":"8f28786efbbcc16aaa5fa58a623970f11be01d7388da126cf784fcccc05f94e3"} Jul 06 15:54:50.098243 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:50.098222 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:50.099186 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:50.099167 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:54:50.233072 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:50.233032 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:50.338935 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:50.338907 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:51.104320 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:51.104280 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" event={"ID":"f8a11536-02d8-4c71-ac6d-b4eb052f119b","Type":"ContainerStarted","Data":"6a84c1315bea9fb346a39ed022299b4d4b1c51e4cf3b6f966791ada8817ee3d6"} Jul 06 15:54:51.127733 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:51.127688 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" podStartSLOduration=1.6052998550000002 podStartE2EDuration="2.127672165s" totalImagesPullingTime="522.37231ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:54:51.125724906 +0000 UTC m=+1712.758276902" watchObservedRunningTime="2026-07-06 15:54:51.127672165 +0000 UTC m=+1712.760224148" Jul 06 15:54:52.026445 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:52.026416 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:52.111549 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:52.111493 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" Jul 06 15:54:54.613257 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:54.613206 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:54.738318 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:54.738281 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96"] Jul 06 15:54:54.738501 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:54.738476 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" podUID="df2aeac3-05fa-4db0-a00e-bb2697587c9e" containerName="istio-proxy" containerID="cri-o://1fcf1b09b950c3192280bcc5d1c49f9e06e34902a24996a817c0cf4dc1989d1c" gracePeriod=30 Jul 06 15:54:55.016509 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:55.016477 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6"] Jul 06 15:54:55.016984 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:55.016952 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" podUID="f8a11536-02d8-4c71-ac6d-b4eb052f119b" containerName="maas-api" containerID="cri-o://6a84c1315bea9fb346a39ed022299b4d4b1c51e4cf3b6f966791ada8817ee3d6" gracePeriod=30 Jul 06 15:54:55.041917 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:55.039806 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" Jul 06 15:54:55.327555 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:55.323699 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" Jul 06 15:54:55.397913 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:55.397889 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a11536-02d8-4c71-ac6d-b4eb052f119b-maas-api-tls\") pod \"f8a11536-02d8-4c71-ac6d-b4eb052f119b\" (UID: \"f8a11536-02d8-4c71-ac6d-b4eb052f119b\") " Jul 06 15:54:55.398061 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:55.397934 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a11536-02d8-4c71-ac6d-b4eb052f119b-openshift-service-ca\") pod \"f8a11536-02d8-4c71-ac6d-b4eb052f119b\" (UID: \"f8a11536-02d8-4c71-ac6d-b4eb052f119b\") " Jul 06 15:54:55.398061 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:55.397984 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s8km\" (UniqueName: \"kubernetes.io/projected/f8a11536-02d8-4c71-ac6d-b4eb052f119b-kube-api-access-5s8km\") pod \"f8a11536-02d8-4c71-ac6d-b4eb052f119b\" (UID: \"f8a11536-02d8-4c71-ac6d-b4eb052f119b\") " Jul 06 15:54:55.398485 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:55.398437 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a11536-02d8-4c71-ac6d-b4eb052f119b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f8a11536-02d8-4c71-ac6d-b4eb052f119b" (UID: "f8a11536-02d8-4c71-ac6d-b4eb052f119b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:54:55.400353 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:55.400324 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a11536-02d8-4c71-ac6d-b4eb052f119b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f8a11536-02d8-4c71-ac6d-b4eb052f119b" (UID: "f8a11536-02d8-4c71-ac6d-b4eb052f119b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:54:55.400416 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:55.400399 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a11536-02d8-4c71-ac6d-b4eb052f119b-kube-api-access-5s8km" (OuterVolumeSpecName: "kube-api-access-5s8km") pod "f8a11536-02d8-4c71-ac6d-b4eb052f119b" (UID: "f8a11536-02d8-4c71-ac6d-b4eb052f119b"). InnerVolumeSpecName "kube-api-access-5s8km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:54:55.499096 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:55.499052 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8a11536-02d8-4c71-ac6d-b4eb052f119b-maas-api-tls\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:54:55.499096 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:55.499091 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8a11536-02d8-4c71-ac6d-b4eb052f119b-openshift-service-ca\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:54:55.499096 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:55.499101 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s8km\" (UniqueName: \"kubernetes.io/projected/f8a11536-02d8-4c71-ac6d-b4eb052f119b-kube-api-access-5s8km\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:54:56.088444 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:56.088407 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:56.142797 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:56.142767 2561 generic.go:356] "Generic (PLEG): container finished" podID="f8a11536-02d8-4c71-ac6d-b4eb052f119b" containerID="6a84c1315bea9fb346a39ed022299b4d4b1c51e4cf3b6f966791ada8817ee3d6" exitCode=0 Jul 06 15:54:56.142959 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:56.142815 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" Jul 06 15:54:56.142959 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:56.142857 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" event={"ID":"f8a11536-02d8-4c71-ac6d-b4eb052f119b","Type":"ContainerDied","Data":"6a84c1315bea9fb346a39ed022299b4d4b1c51e4cf3b6f966791ada8817ee3d6"} Jul 06 15:54:56.142959 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:56.142917 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6" event={"ID":"f8a11536-02d8-4c71-ac6d-b4eb052f119b","Type":"ContainerDied","Data":"8f28786efbbcc16aaa5fa58a623970f11be01d7388da126cf784fcccc05f94e3"} Jul 06 15:54:56.142959 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:56.142941 2561 scope.go:117] "RemoveContainer" containerID="6a84c1315bea9fb346a39ed022299b4d4b1c51e4cf3b6f966791ada8817ee3d6" Jul 06 15:54:56.153141 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:56.153126 2561 scope.go:117] "RemoveContainer" containerID="6a84c1315bea9fb346a39ed022299b4d4b1c51e4cf3b6f966791ada8817ee3d6" Jul 06 15:54:56.153399 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:54:56.153376 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a84c1315bea9fb346a39ed022299b4d4b1c51e4cf3b6f966791ada8817ee3d6\": container with ID starting with 6a84c1315bea9fb346a39ed022299b4d4b1c51e4cf3b6f966791ada8817ee3d6 not found: ID does not exist" containerID="6a84c1315bea9fb346a39ed022299b4d4b1c51e4cf3b6f966791ada8817ee3d6" Jul 06 15:54:56.153450 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:56.153404 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a84c1315bea9fb346a39ed022299b4d4b1c51e4cf3b6f966791ada8817ee3d6"} err="failed to get container status \"6a84c1315bea9fb346a39ed022299b4d4b1c51e4cf3b6f966791ada8817ee3d6\": rpc error: code = NotFound desc = could not find container \"6a84c1315bea9fb346a39ed022299b4d4b1c51e4cf3b6f966791ada8817ee3d6\": container with ID starting with 6a84c1315bea9fb346a39ed022299b4d4b1c51e4cf3b6f966791ada8817ee3d6 not found: ID does not exist" Jul 06 15:54:56.169501 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:56.169473 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6"] Jul 06 15:54:56.179413 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:56.179386 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-0ae7973b-5bf89688fc-nk4p6"] Jul 06 15:54:56.363801 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:56.363718 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:54:56.800574 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:54:56.800544 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a11536-02d8-4c71-ac6d-b4eb052f119b" path="/var/lib/kubelet/pods/f8a11536-02d8-4c71-ac6d-b4eb052f119b/volumes" Jul 06 15:55:00.412368 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.412343 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:55:00.553284 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.553256 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-workload-socket\") pod \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " Jul 06 15:55:00.553442 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.553289 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-token\") pod \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " Jul 06 15:55:00.553442 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.553310 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-data\") pod \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " Jul 06 15:55:00.553442 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.553325 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-envoy\") pod \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " Jul 06 15:55:00.553442 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.553384 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istiod-ca-cert\") pod \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " Jul 06 15:55:00.553442 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.553413 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-workload-certs\") pod \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " Jul 06 15:55:00.553753 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.553449 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjw5z\" (UniqueName: \"kubernetes.io/projected/df2aeac3-05fa-4db0-a00e-bb2697587c9e-kube-api-access-mjw5z\") pod \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " Jul 06 15:55:00.553753 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.553507 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-credential-socket\") pod \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " Jul 06 15:55:00.553753 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.553554 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-podinfo\") pod \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\" (UID: \"df2aeac3-05fa-4db0-a00e-bb2697587c9e\") " Jul 06 15:55:00.553753 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.553597 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "df2aeac3-05fa-4db0-a00e-bb2697587c9e" (UID: "df2aeac3-05fa-4db0-a00e-bb2697587c9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:00.553965 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.553875 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-workload-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:00.554150 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.554116 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "df2aeac3-05fa-4db0-a00e-bb2697587c9e" (UID: "df2aeac3-05fa-4db0-a00e-bb2697587c9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:00.554317 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.554295 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-data" (OuterVolumeSpecName: "istio-data") pod "df2aeac3-05fa-4db0-a00e-bb2697587c9e" (UID: "df2aeac3-05fa-4db0-a00e-bb2697587c9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:00.554461 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.554396 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "df2aeac3-05fa-4db0-a00e-bb2697587c9e" (UID: "df2aeac3-05fa-4db0-a00e-bb2697587c9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:00.554767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.554744 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "df2aeac3-05fa-4db0-a00e-bb2697587c9e" (UID: "df2aeac3-05fa-4db0-a00e-bb2697587c9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:55:00.555805 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.555773 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2aeac3-05fa-4db0-a00e-bb2697587c9e-kube-api-access-mjw5z" (OuterVolumeSpecName: "kube-api-access-mjw5z") pod "df2aeac3-05fa-4db0-a00e-bb2697587c9e" (UID: "df2aeac3-05fa-4db0-a00e-bb2697587c9e"). InnerVolumeSpecName "kube-api-access-mjw5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:55:00.555954 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.555934 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "df2aeac3-05fa-4db0-a00e-bb2697587c9e" (UID: "df2aeac3-05fa-4db0-a00e-bb2697587c9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:00.556104 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.556084 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-token" (OuterVolumeSpecName: "istio-token") pod "df2aeac3-05fa-4db0-a00e-bb2697587c9e" (UID: "df2aeac3-05fa-4db0-a00e-bb2697587c9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:55:00.556104 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.556092 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "df2aeac3-05fa-4db0-a00e-bb2697587c9e" (UID: "df2aeac3-05fa-4db0-a00e-bb2697587c9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 15:55:00.654867 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.654830 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-workload-certs\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:00.654867 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.654864 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjw5z\" (UniqueName: \"kubernetes.io/projected/df2aeac3-05fa-4db0-a00e-bb2697587c9e-kube-api-access-mjw5z\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:00.655039 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.654874 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-credential-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:00.655039 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.654883 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-podinfo\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:00.655039 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.654890 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-token\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:00.655039 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.654898 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-data\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:00.655039 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.654905 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istio-envoy\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:00.655039 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:00.654912 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df2aeac3-05fa-4db0-a00e-bb2697587c9e-istiod-ca-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:01.174210 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:01.174127 2561 generic.go:356] "Generic (PLEG): container finished" podID="df2aeac3-05fa-4db0-a00e-bb2697587c9e" containerID="1fcf1b09b950c3192280bcc5d1c49f9e06e34902a24996a817c0cf4dc1989d1c" exitCode=0 Jul 06 15:55:01.174210 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:01.174189 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" Jul 06 15:55:01.174210 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:01.174187 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" event={"ID":"df2aeac3-05fa-4db0-a00e-bb2697587c9e","Type":"ContainerDied","Data":"1fcf1b09b950c3192280bcc5d1c49f9e06e34902a24996a817c0cf4dc1989d1c"} Jul 06 15:55:01.174471 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:01.174283 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96" event={"ID":"df2aeac3-05fa-4db0-a00e-bb2697587c9e","Type":"ContainerDied","Data":"4bd1ad7e39f93f5176e2a06b9df4b1bb0ff6f01b77eb4ca74471a6e1c442693b"} Jul 06 15:55:01.174471 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:01.174314 2561 scope.go:117] "RemoveContainer" containerID="1fcf1b09b950c3192280bcc5d1c49f9e06e34902a24996a817c0cf4dc1989d1c" Jul 06 15:55:01.183092 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:01.183076 2561 scope.go:117] "RemoveContainer" containerID="1fcf1b09b950c3192280bcc5d1c49f9e06e34902a24996a817c0cf4dc1989d1c" Jul 06 15:55:01.183345 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:55:01.183327 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fcf1b09b950c3192280bcc5d1c49f9e06e34902a24996a817c0cf4dc1989d1c\": container with ID starting with 1fcf1b09b950c3192280bcc5d1c49f9e06e34902a24996a817c0cf4dc1989d1c not found: ID does not exist" containerID="1fcf1b09b950c3192280bcc5d1c49f9e06e34902a24996a817c0cf4dc1989d1c" Jul 06 15:55:01.183397 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:01.183354 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fcf1b09b950c3192280bcc5d1c49f9e06e34902a24996a817c0cf4dc1989d1c"} err="failed to get container status \"1fcf1b09b950c3192280bcc5d1c49f9e06e34902a24996a817c0cf4dc1989d1c\": rpc error: code = NotFound desc = could not find container \"1fcf1b09b950c3192280bcc5d1c49f9e06e34902a24996a817c0cf4dc1989d1c\": container with ID starting with 1fcf1b09b950c3192280bcc5d1c49f9e06e34902a24996a817c0cf4dc1989d1c not found: ID does not exist" Jul 06 15:55:01.198352 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:01.198331 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96"] Jul 06 15:55:01.205584 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:01.205564 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0ae7973b-openshift-default-7c67f56b8c-kvf96"] Jul 06 15:55:01.849686 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:01.849652 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:02.012117 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.012090 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:02.240234 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.240196 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46"] Jul 06 15:55:02.240710 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.240691 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a11536-02d8-4c71-ac6d-b4eb052f119b" containerName="maas-api" Jul 06 15:55:02.240710 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.240710 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a11536-02d8-4c71-ac6d-b4eb052f119b" containerName="maas-api" Jul 06 15:55:02.240869 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.240735 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df2aeac3-05fa-4db0-a00e-bb2697587c9e" containerName="istio-proxy" Jul 06 15:55:02.240869 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.240743 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="df2aeac3-05fa-4db0-a00e-bb2697587c9e" containerName="istio-proxy" Jul 06 15:55:02.240869 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.240840 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="df2aeac3-05fa-4db0-a00e-bb2697587c9e" containerName="istio-proxy" Jul 06 15:55:02.240869 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.240852 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a11536-02d8-4c71-ac6d-b4eb052f119b" containerName="maas-api" Jul 06 15:55:02.245879 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.245861 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.249558 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.249521 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-08adabf9-gw-openshift-default-dockercfg-lp2jp\"" Jul 06 15:55:02.256370 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.256350 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:02.265200 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.265175 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46"] Jul 06 15:55:02.368905 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.368876 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-credential-socket\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.369078 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.368930 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-data\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.369078 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.368987 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-workload-certs\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.369078 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.369030 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-podinfo\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.369078 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.369050 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-token\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.369232 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.369082 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-envoy\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.369232 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.369150 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccqqg\" (UniqueName: \"kubernetes.io/projected/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-kube-api-access-ccqqg\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.369232 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.369176 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-workload-socket\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.369232 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.369197 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istiod-ca-cert\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.401701 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.401671 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:02.470700 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.470672 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-workload-certs\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.470810 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.470709 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-podinfo\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.470810 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.470733 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-token\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.470810 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.470772 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-envoy\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.470948 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.470844 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccqqg\" (UniqueName: \"kubernetes.io/projected/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-kube-api-access-ccqqg\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.470948 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.470879 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-workload-socket\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.470948 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.470911 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istiod-ca-cert\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.471118 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.470959 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-credential-socket\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.471118 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.471022 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-data\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.471118 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.471095 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-workload-certs\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.471449 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.471424 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-data\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.471514 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.471455 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-workload-socket\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.471639 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.471621 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istiod-ca-cert\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.471688 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.471669 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-credential-socket\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.473237 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.473211 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-envoy\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.473392 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.473377 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-podinfo\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.484800 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.484773 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-token\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.484964 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.484945 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccqqg\" (UniqueName: \"kubernetes.io/projected/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-kube-api-access-ccqqg\") pod \"e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.560120 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.560064 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:02.707506 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.707476 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46"] Jul 06 15:55:02.708668 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:55:02.708635 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded6bb1c_79a1_4e1a_8ba7_d6fd18a94fea.slice/crio-e8b64a09f274e859575ac651e7a545cff2f1e2012a16f266e9aac132dbadd532 WatchSource:0}: Error finding container e8b64a09f274e859575ac651e7a545cff2f1e2012a16f266e9aac132dbadd532: Status 404 returned error can't find the container with id e8b64a09f274e859575ac651e7a545cff2f1e2012a16f266e9aac132dbadd532 Jul 06 15:55:02.711271 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.711233 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:55:02.711390 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.711314 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:55:02.711390 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.711356 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:55:02.800642 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:02.800606 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df2aeac3-05fa-4db0-a00e-bb2697587c9e" path="/var/lib/kubelet/pods/df2aeac3-05fa-4db0-a00e-bb2697587c9e/volumes" Jul 06 15:55:03.186179 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.186148 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" event={"ID":"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea","Type":"ContainerStarted","Data":"5563c52aca1b1fea2cbdff718a377a8c6b1c1883497c09be9bdd608588a490f3"} Jul 06 15:55:03.186691 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.186212 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" event={"ID":"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea","Type":"ContainerStarted","Data":"e8b64a09f274e859575ac651e7a545cff2f1e2012a16f266e9aac132dbadd532"} Jul 06 15:55:03.216566 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.216502 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" podStartSLOduration=1.216486715 podStartE2EDuration="1.216486715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:55:03.214189789 +0000 UTC m=+1724.846741772" watchObservedRunningTime="2026-07-06 15:55:03.216486715 +0000 UTC m=+1724.849038697" Jul 06 15:55:03.561277 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.561244 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:03.566670 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.566645 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:03.691453 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.691418 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:03.751915 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.751874 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn"] Jul 06 15:55:03.755776 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.755756 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" Jul 06 15:55:03.758710 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.758691 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rszzk\"" Jul 06 15:55:03.758710 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.758707 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-migrate-08adabf9-serving-cert\"" Jul 06 15:55:03.769322 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.769300 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn"] Jul 06 15:55:03.814290 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.814213 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:03.886685 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.886655 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w96rb\" (UniqueName: \"kubernetes.io/projected/840c3311-2a4a-4ffe-944b-dec319b08995-kube-api-access-w96rb\") pod \"maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn\" (UID: \"840c3311-2a4a-4ffe-944b-dec319b08995\") " pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" Jul 06 15:55:03.886837 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.886697 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840c3311-2a4a-4ffe-944b-dec319b08995-openshift-service-ca\") pod \"maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn\" (UID: \"840c3311-2a4a-4ffe-944b-dec319b08995\") " pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" Jul 06 15:55:03.886837 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.886787 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840c3311-2a4a-4ffe-944b-dec319b08995-maas-api-tls\") pod \"maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn\" (UID: \"840c3311-2a4a-4ffe-944b-dec319b08995\") " pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" Jul 06 15:55:03.987816 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.987780 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w96rb\" (UniqueName: \"kubernetes.io/projected/840c3311-2a4a-4ffe-944b-dec319b08995-kube-api-access-w96rb\") pod \"maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn\" (UID: \"840c3311-2a4a-4ffe-944b-dec319b08995\") " pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" Jul 06 15:55:03.987963 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.987832 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840c3311-2a4a-4ffe-944b-dec319b08995-openshift-service-ca\") pod \"maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn\" (UID: \"840c3311-2a4a-4ffe-944b-dec319b08995\") " pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" Jul 06 15:55:03.987963 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.987879 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840c3311-2a4a-4ffe-944b-dec319b08995-maas-api-tls\") pod \"maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn\" (UID: \"840c3311-2a4a-4ffe-944b-dec319b08995\") " pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" Jul 06 15:55:03.988085 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:55:03.988064 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-migrate-08adabf9-serving-cert: secret "maas-api-e2e-migrate-08adabf9-serving-cert" not found Jul 06 15:55:03.988150 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:55:03.988139 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/840c3311-2a4a-4ffe-944b-dec319b08995-maas-api-tls podName:840c3311-2a4a-4ffe-944b-dec319b08995 nodeName:}" failed. No retries permitted until 2026-07-06 15:55:04.488123889 +0000 UTC m=+1726.120675853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/840c3311-2a4a-4ffe-944b-dec319b08995-maas-api-tls") pod "maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" (UID: "840c3311-2a4a-4ffe-944b-dec319b08995") : secret "maas-api-e2e-migrate-08adabf9-serving-cert" not found Jul 06 15:55:03.988498 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.988480 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840c3311-2a4a-4ffe-944b-dec319b08995-openshift-service-ca\") pod \"maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn\" (UID: \"840c3311-2a4a-4ffe-944b-dec319b08995\") " pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" Jul 06 15:55:03.998763 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:03.998737 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w96rb\" (UniqueName: \"kubernetes.io/projected/840c3311-2a4a-4ffe-944b-dec319b08995-kube-api-access-w96rb\") pod \"maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn\" (UID: \"840c3311-2a4a-4ffe-944b-dec319b08995\") " pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" Jul 06 15:55:04.191671 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:04.191589 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:04.192780 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:04.192758 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:04.432440 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:04.432237 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:04.493044 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:04.493014 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840c3311-2a4a-4ffe-944b-dec319b08995-maas-api-tls\") pod \"maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn\" (UID: \"840c3311-2a4a-4ffe-944b-dec319b08995\") " pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" Jul 06 15:55:04.495831 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:04.495807 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840c3311-2a4a-4ffe-944b-dec319b08995-maas-api-tls\") pod \"maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn\" (UID: \"840c3311-2a4a-4ffe-944b-dec319b08995\") " pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" Jul 06 15:55:04.549630 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:04.549605 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:04.670075 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:04.670032 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" Jul 06 15:55:04.675929 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:04.675894 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:04.849143 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:04.849117 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn"] Jul 06 15:55:04.850910 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:55:04.850870 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840c3311_2a4a_4ffe_944b_dec319b08995.slice/crio-a30952b64246ef4e7f228a5636e2dc0f5ff03f0c3d2bd7e2ccf1c790809f465f WatchSource:0}: Error finding container a30952b64246ef4e7f228a5636e2dc0f5ff03f0c3d2bd7e2ccf1c790809f465f: Status 404 returned error can't find the container with id a30952b64246ef4e7f228a5636e2dc0f5ff03f0c3d2bd7e2ccf1c790809f465f Jul 06 15:55:05.198759 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:05.198666 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" event={"ID":"840c3311-2a4a-4ffe-944b-dec319b08995","Type":"ContainerStarted","Data":"a30952b64246ef4e7f228a5636e2dc0f5ff03f0c3d2bd7e2ccf1c790809f465f"} Jul 06 15:55:05.339592 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:05.339552 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:06.206203 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:06.206163 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" event={"ID":"840c3311-2a4a-4ffe-944b-dec319b08995","Type":"ContainerStarted","Data":"9488f606b0a3e59198d93e70b24b6be8a2c443c85471ebf12dee4453d82df93e"} Jul 06 15:55:06.233581 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:06.233510 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" podStartSLOduration=2.742843731 podStartE2EDuration="3.233496279s" totalImagesPullingTime="490.652548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:55:06.229897195 +0000 UTC m=+1727.862449177" watchObservedRunningTime="2026-07-06 15:55:06.233496279 +0000 UTC m=+1727.866048262" Jul 06 15:55:07.211783 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:07.211752 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" Jul 06 15:55:08.171322 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:08.171289 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:08.284927 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:08.284878 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46"] Jul 06 15:55:08.285330 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:08.285076 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" podUID="ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea" containerName="istio-proxy" containerID="cri-o://5563c52aca1b1fea2cbdff718a377a8c6b1c1883497c09be9bdd608588a490f3" gracePeriod=30 Jul 06 15:55:08.310842 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:08.310715 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:08.447134 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:08.447045 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:08.584105 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:08.584074 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn"] Jul 06 15:55:08.612554 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:08.611007 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:09.226711 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:09.226629 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" podUID="840c3311-2a4a-4ffe-944b-dec319b08995" containerName="maas-api" containerID="cri-o://9488f606b0a3e59198d93e70b24b6be8a2c443c85471ebf12dee4453d82df93e" gracePeriod=30 Jul 06 15:55:09.504607 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:09.504582 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" Jul 06 15:55:09.650250 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:09.650219 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840c3311-2a4a-4ffe-944b-dec319b08995-maas-api-tls\") pod \"840c3311-2a4a-4ffe-944b-dec319b08995\" (UID: \"840c3311-2a4a-4ffe-944b-dec319b08995\") " Jul 06 15:55:09.650422 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:09.650338 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840c3311-2a4a-4ffe-944b-dec319b08995-openshift-service-ca\") pod \"840c3311-2a4a-4ffe-944b-dec319b08995\" (UID: \"840c3311-2a4a-4ffe-944b-dec319b08995\") " Jul 06 15:55:09.650422 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:09.650398 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w96rb\" (UniqueName: \"kubernetes.io/projected/840c3311-2a4a-4ffe-944b-dec319b08995-kube-api-access-w96rb\") pod \"840c3311-2a4a-4ffe-944b-dec319b08995\" (UID: \"840c3311-2a4a-4ffe-944b-dec319b08995\") " Jul 06 15:55:09.650759 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:09.650733 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/840c3311-2a4a-4ffe-944b-dec319b08995-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "840c3311-2a4a-4ffe-944b-dec319b08995" (UID: "840c3311-2a4a-4ffe-944b-dec319b08995"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:55:09.652581 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:09.652521 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/840c3311-2a4a-4ffe-944b-dec319b08995-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "840c3311-2a4a-4ffe-944b-dec319b08995" (UID: "840c3311-2a4a-4ffe-944b-dec319b08995"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:55:09.652581 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:09.652576 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840c3311-2a4a-4ffe-944b-dec319b08995-kube-api-access-w96rb" (OuterVolumeSpecName: "kube-api-access-w96rb") pod "840c3311-2a4a-4ffe-944b-dec319b08995" (UID: "840c3311-2a4a-4ffe-944b-dec319b08995"). InnerVolumeSpecName "kube-api-access-w96rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:55:09.752029 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:09.751954 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/840c3311-2a4a-4ffe-944b-dec319b08995-openshift-service-ca\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:09.752029 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:09.751977 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w96rb\" (UniqueName: \"kubernetes.io/projected/840c3311-2a4a-4ffe-944b-dec319b08995-kube-api-access-w96rb\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:09.752029 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:09.751991 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/840c3311-2a4a-4ffe-944b-dec319b08995-maas-api-tls\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:10.235425 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:10.235388 2561 generic.go:356] "Generic (PLEG): container finished" podID="840c3311-2a4a-4ffe-944b-dec319b08995" containerID="9488f606b0a3e59198d93e70b24b6be8a2c443c85471ebf12dee4453d82df93e" exitCode=0 Jul 06 15:55:10.235425 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:10.235428 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" Jul 06 15:55:10.235710 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:10.235473 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" event={"ID":"840c3311-2a4a-4ffe-944b-dec319b08995","Type":"ContainerDied","Data":"9488f606b0a3e59198d93e70b24b6be8a2c443c85471ebf12dee4453d82df93e"} Jul 06 15:55:10.235710 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:10.235554 2561 scope.go:117] "RemoveContainer" containerID="9488f606b0a3e59198d93e70b24b6be8a2c443c85471ebf12dee4453d82df93e" Jul 06 15:55:10.235710 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:10.235681 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn" event={"ID":"840c3311-2a4a-4ffe-944b-dec319b08995","Type":"ContainerDied","Data":"a30952b64246ef4e7f228a5636e2dc0f5ff03f0c3d2bd7e2ccf1c790809f465f"} Jul 06 15:55:10.245928 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:10.245900 2561 scope.go:117] "RemoveContainer" containerID="9488f606b0a3e59198d93e70b24b6be8a2c443c85471ebf12dee4453d82df93e" Jul 06 15:55:10.250253 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:55:10.246808 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9488f606b0a3e59198d93e70b24b6be8a2c443c85471ebf12dee4453d82df93e\": container with ID starting with 9488f606b0a3e59198d93e70b24b6be8a2c443c85471ebf12dee4453d82df93e not found: ID does not exist" containerID="9488f606b0a3e59198d93e70b24b6be8a2c443c85471ebf12dee4453d82df93e" Jul 06 15:55:10.250253 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:10.246845 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9488f606b0a3e59198d93e70b24b6be8a2c443c85471ebf12dee4453d82df93e"} err="failed to get container status \"9488f606b0a3e59198d93e70b24b6be8a2c443c85471ebf12dee4453d82df93e\": rpc error: code = NotFound desc = could not find container \"9488f606b0a3e59198d93e70b24b6be8a2c443c85471ebf12dee4453d82df93e\": container with ID starting with 9488f606b0a3e59198d93e70b24b6be8a2c443c85471ebf12dee4453d82df93e not found: ID does not exist" Jul 06 15:55:10.259997 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:10.259962 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn"] Jul 06 15:55:10.266723 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:10.266705 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-migrate-08adabf9-565f69cf45-2tjgn"] Jul 06 15:55:10.799986 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:10.799951 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="840c3311-2a4a-4ffe-944b-dec319b08995" path="/var/lib/kubelet/pods/840c3311-2a4a-4ffe-944b-dec319b08995/volumes" Jul 06 15:55:11.751190 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:11.751156 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:14.868480 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:14.868449 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:14.978086 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:14.978054 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:15.084995 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.084967 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx"] Jul 06 15:55:15.087062 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.085623 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="840c3311-2a4a-4ffe-944b-dec319b08995" containerName="maas-api" Jul 06 15:55:15.087062 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.085640 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="840c3311-2a4a-4ffe-944b-dec319b08995" containerName="maas-api" Jul 06 15:55:15.087062 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.085742 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="840c3311-2a4a-4ffe-944b-dec319b08995" containerName="maas-api" Jul 06 15:55:15.089112 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.089092 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.092223 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.092201 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-47b69c3c-openshift-default-dockercfg-k9q49\"" Jul 06 15:55:15.104332 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.104311 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx"] Jul 06 15:55:15.137627 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.137521 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:15.207373 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.206452 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-data\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.207373 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.206498 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-podinfo\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.207373 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.206580 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-credential-socket\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.207373 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.206608 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-workload-certs\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.207373 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.206633 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-token\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.207373 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.206695 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istiod-ca-cert\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.207373 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.206766 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-envoy\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.207373 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.206805 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwwcs\" (UniqueName: \"kubernetes.io/projected/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-kube-api-access-gwwcs\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.207373 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.206844 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-workload-socket\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.230759 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.230734 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:15.267096 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.267069 2561 generic.go:356] "Generic (PLEG): container finished" podID="ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea" containerID="5563c52aca1b1fea2cbdff718a377a8c6b1c1883497c09be9bdd608588a490f3" exitCode=0 Jul 06 15:55:15.267239 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.267189 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" event={"ID":"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea","Type":"ContainerDied","Data":"5563c52aca1b1fea2cbdff718a377a8c6b1c1883497c09be9bdd608588a490f3"} Jul 06 15:55:15.307507 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.307430 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-envoy\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.307507 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.307478 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwwcs\" (UniqueName: \"kubernetes.io/projected/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-kube-api-access-gwwcs\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.307934 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.307897 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-workload-socket\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.308080 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.307995 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-workload-socket\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.308217 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.308156 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-data\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.308371 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.308353 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-podinfo\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.308467 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.308396 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-credential-socket\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.308467 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.308419 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-workload-certs\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.308467 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.308445 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-token\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.308739 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.308507 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istiod-ca-cert\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.308739 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.308605 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-data\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.309043 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.308996 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-workload-certs\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.309322 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.309296 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-credential-socket\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.309738 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.309716 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istiod-ca-cert\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.311784 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.311743 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-envoy\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.312201 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.312180 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-podinfo\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.317810 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.317776 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-token\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.318084 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.318061 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwwcs\" (UniqueName: \"kubernetes.io/projected/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-kube-api-access-gwwcs\") pod \"e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.367616 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.367591 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:15.400947 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.400868 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:15.510870 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.510839 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-data\") pod \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " Jul 06 15:55:15.511860 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.511837 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccqqg\" (UniqueName: \"kubernetes.io/projected/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-kube-api-access-ccqqg\") pod \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " Jul 06 15:55:15.512089 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.512074 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-workload-certs\") pod \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " Jul 06 15:55:15.512228 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.512216 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-podinfo\") pod \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " Jul 06 15:55:15.512350 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.512338 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-envoy\") pod \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " Jul 06 15:55:15.512465 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.512453 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-credential-socket\") pod \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " Jul 06 15:55:15.512657 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.511774 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-data" (OuterVolumeSpecName: "istio-data") pod "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea" (UID: "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:15.512797 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.512780 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-workload-socket\") pod \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " Jul 06 15:55:15.513187 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.513147 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea" (UID: "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:15.513283 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.513222 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea" (UID: "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:15.514251 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.513375 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-token\") pod \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " Jul 06 15:55:15.514251 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.513432 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istiod-ca-cert\") pod \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\" (UID: \"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea\") " Jul 06 15:55:15.514251 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.513821 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-workload-certs\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:15.514251 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.513840 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-credential-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:15.514251 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.513852 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-data\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:15.514980 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.514755 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea" (UID: "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:55:15.515146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.515129 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea" (UID: "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:15.516604 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.516564 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea" (UID: "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:15.518092 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.518065 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea" (UID: "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 15:55:15.518194 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.518176 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-token" (OuterVolumeSpecName: "istio-token") pod "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea" (UID: "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:55:15.518264 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.518188 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-kube-api-access-ccqqg" (OuterVolumeSpecName: "kube-api-access-ccqqg") pod "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea" (UID: "ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea"). InnerVolumeSpecName "kube-api-access-ccqqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:55:15.559696 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.559665 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx"] Jul 06 15:55:15.560365 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:55:15.560341 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931bed4e_bb0c_45d9_8c0e_12b102c53f3d.slice/crio-759678a9bd2405e91dbae54af67a778a2754d94f8ba2b462715e35c9ca3c1416 WatchSource:0}: Error finding container 759678a9bd2405e91dbae54af67a778a2754d94f8ba2b462715e35c9ca3c1416: Status 404 returned error can't find the container with id 759678a9bd2405e91dbae54af67a778a2754d94f8ba2b462715e35c9ca3c1416 Jul 06 15:55:15.562952 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.562911 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:55:15.563064 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.562986 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:55:15.563064 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.563036 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:55:15.615486 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.615455 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-podinfo\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:15.615663 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.615489 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-envoy\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:15.615663 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.615505 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-workload-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:15.615663 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.615519 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istio-token\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:15.615663 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.615557 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-istiod-ca-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:15.615663 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:15.615570 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccqqg\" (UniqueName: \"kubernetes.io/projected/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea-kube-api-access-ccqqg\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:16.274627 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.274589 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" event={"ID":"931bed4e-bb0c-45d9-8c0e-12b102c53f3d","Type":"ContainerStarted","Data":"e2b013208ec0d9b059dda1db81d0bef25a5a74662f5ae9b416e39607c063d96a"} Jul 06 15:55:16.275005 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.274649 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" event={"ID":"931bed4e-bb0c-45d9-8c0e-12b102c53f3d","Type":"ContainerStarted","Data":"759678a9bd2405e91dbae54af67a778a2754d94f8ba2b462715e35c9ca3c1416"} Jul 06 15:55:16.275964 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.275947 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" Jul 06 15:55:16.276083 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.275945 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46" event={"ID":"ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea","Type":"ContainerDied","Data":"e8b64a09f274e859575ac651e7a545cff2f1e2012a16f266e9aac132dbadd532"} Jul 06 15:55:16.276144 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.276102 2561 scope.go:117] "RemoveContainer" containerID="5563c52aca1b1fea2cbdff718a377a8c6b1c1883497c09be9bdd608588a490f3" Jul 06 15:55:16.297600 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.297554 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" podStartSLOduration=1.297516248 podStartE2EDuration="1.297516248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:55:16.296146315 +0000 UTC m=+1737.928698298" watchObservedRunningTime="2026-07-06 15:55:16.297516248 +0000 UTC m=+1737.930068231" Jul 06 15:55:16.313776 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.313750 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46"] Jul 06 15:55:16.320573 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.320548 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-08adabf9-gw-openshift-default-d9b9487b7-btx46"] Jul 06 15:55:16.391696 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.391666 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:16.401683 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.401632 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:16.407504 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.407480 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:16.472884 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.472858 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:16.474159 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.474139 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks"] Jul 06 15:55:16.474601 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.474588 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea" containerName="istio-proxy" Jul 06 15:55:16.474664 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.474602 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea" containerName="istio-proxy" Jul 06 15:55:16.474708 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.474680 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea" containerName="istio-proxy" Jul 06 15:55:16.479095 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.479080 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:16.482063 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.482046 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-47b69c3c-serving-cert\"" Jul 06 15:55:16.482182 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.482068 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rszzk\"" Jul 06 15:55:16.499354 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.499326 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks"] Jul 06 15:55:16.523997 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.523967 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-maas-api-tls\") pod \"maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks\" (UID: \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\") " pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:16.524357 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.524228 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-openshift-service-ca\") pod \"maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks\" (UID: \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\") " pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:16.524357 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.524314 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqmxt\" (UniqueName: \"kubernetes.io/projected/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-kube-api-access-bqmxt\") pod \"maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks\" (UID: \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\") " pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:16.585065 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.585037 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:16.625282 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.625236 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-openshift-service-ca\") pod \"maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks\" (UID: \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\") " pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:16.625446 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.625296 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqmxt\" (UniqueName: \"kubernetes.io/projected/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-kube-api-access-bqmxt\") pod \"maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks\" (UID: \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\") " pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:16.625446 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.625325 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-maas-api-tls\") pod \"maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks\" (UID: \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\") " pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:16.625446 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:55:16.625441 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-47b69c3c-serving-cert: secret "maas-api-e2e-ait-47b69c3c-serving-cert" not found Jul 06 15:55:16.625605 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:55:16.625506 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-maas-api-tls podName:6f4918e0-a836-4f64-85ec-0bb6252f1a5e nodeName:}" failed. No retries permitted until 2026-07-06 15:55:17.125487506 +0000 UTC m=+1738.758039481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-maas-api-tls") pod "maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" (UID: "6f4918e0-a836-4f64-85ec-0bb6252f1a5e") : secret "maas-api-e2e-ait-47b69c3c-serving-cert" not found Jul 06 15:55:16.626064 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.626036 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-openshift-service-ca\") pod \"maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks\" (UID: \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\") " pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:16.640280 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.640244 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqmxt\" (UniqueName: \"kubernetes.io/projected/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-kube-api-access-bqmxt\") pod \"maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks\" (UID: \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\") " pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:16.802086 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:16.802012 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea" path="/var/lib/kubelet/pods/ded6bb1c-79a1-4e1a-8ba7-d6fd18a94fea/volumes" Jul 06 15:55:17.126308 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:17.126221 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:17.129472 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:17.129453 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-maas-api-tls\") pod \"maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks\" (UID: \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\") " pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:17.131930 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:17.131906 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-maas-api-tls\") pod \"maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks\" (UID: \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\") " pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:17.283898 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:17.283865 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:17.284860 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:17.284842 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:17.387951 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:17.387832 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:17.397488 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:17.397461 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:17.485446 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:17.485420 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:17.577232 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:17.577206 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks"] Jul 06 15:55:17.578893 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:55:17.578866 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4918e0_a836_4f64_85ec_0bb6252f1a5e.slice/crio-ad3fc9833472364fa84789f9678f9810a51013259631796bd31252854a2725ad WatchSource:0}: Error finding container ad3fc9833472364fa84789f9678f9810a51013259631796bd31252854a2725ad: Status 404 returned error can't find the container with id ad3fc9833472364fa84789f9678f9810a51013259631796bd31252854a2725ad Jul 06 15:55:18.210111 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:18.210031 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:18.290667 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:18.290628 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" event={"ID":"6f4918e0-a836-4f64-85ec-0bb6252f1a5e","Type":"ContainerStarted","Data":"ac8b8f28ae230cd8e6ce6692d8cfa9bbdc9ae7fb167f99e70b0671117f5c6163"} Jul 06 15:55:18.291026 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:18.290683 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" event={"ID":"6f4918e0-a836-4f64-85ec-0bb6252f1a5e","Type":"ContainerStarted","Data":"ad3fc9833472364fa84789f9678f9810a51013259631796bd31252854a2725ad"} Jul 06 15:55:18.291026 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:18.290723 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:18.317443 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:18.317399 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" podStartSLOduration=1.994918035 podStartE2EDuration="2.317384893s" totalImagesPullingTime="322.466858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:55:18.314807504 +0000 UTC m=+1739.947359492" watchObservedRunningTime="2026-07-06 15:55:18.317384893 +0000 UTC m=+1739.949936879" Jul 06 15:55:18.503853 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:18.503824 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:21.556081 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:21.556045 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:22.086695 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.086664 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks"] Jul 06 15:55:22.086909 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.086882 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" podUID="6f4918e0-a836-4f64-85ec-0bb6252f1a5e" containerName="maas-api" containerID="cri-o://ac8b8f28ae230cd8e6ce6692d8cfa9bbdc9ae7fb167f99e70b0671117f5c6163" gracePeriod=30 Jul 06 15:55:22.092679 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.092658 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:22.120135 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.120055 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:22.323552 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.321304 2561 generic.go:356] "Generic (PLEG): container finished" podID="6f4918e0-a836-4f64-85ec-0bb6252f1a5e" containerID="ac8b8f28ae230cd8e6ce6692d8cfa9bbdc9ae7fb167f99e70b0671117f5c6163" exitCode=0 Jul 06 15:55:22.323552 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.321370 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" event={"ID":"6f4918e0-a836-4f64-85ec-0bb6252f1a5e","Type":"ContainerDied","Data":"ac8b8f28ae230cd8e6ce6692d8cfa9bbdc9ae7fb167f99e70b0671117f5c6163"} Jul 06 15:55:22.388347 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.388325 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:22.475949 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.475920 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqmxt\" (UniqueName: \"kubernetes.io/projected/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-kube-api-access-bqmxt\") pod \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\" (UID: \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\") " Jul 06 15:55:22.476120 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.475977 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-openshift-service-ca\") pod \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\" (UID: \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\") " Jul 06 15:55:22.476120 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.475996 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-maas-api-tls\") pod \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\" (UID: \"6f4918e0-a836-4f64-85ec-0bb6252f1a5e\") " Jul 06 15:55:22.476378 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.476352 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f4918e0-a836-4f64-85ec-0bb6252f1a5e" (UID: "6f4918e0-a836-4f64-85ec-0bb6252f1a5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:55:22.478123 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.478095 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f4918e0-a836-4f64-85ec-0bb6252f1a5e" (UID: "6f4918e0-a836-4f64-85ec-0bb6252f1a5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:55:22.478243 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.478133 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-kube-api-access-bqmxt" (OuterVolumeSpecName: "kube-api-access-bqmxt") pod "6f4918e0-a836-4f64-85ec-0bb6252f1a5e" (UID: "6f4918e0-a836-4f64-85ec-0bb6252f1a5e"). InnerVolumeSpecName "kube-api-access-bqmxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:55:22.577606 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.577569 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqmxt\" (UniqueName: \"kubernetes.io/projected/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-kube-api-access-bqmxt\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:22.577606 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.577599 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-openshift-service-ca\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:22.577606 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:22.577612 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4918e0-a836-4f64-85ec-0bb6252f1a5e-maas-api-tls\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:23.330560 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:23.330516 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" Jul 06 15:55:23.330764 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:23.330503 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks" event={"ID":"6f4918e0-a836-4f64-85ec-0bb6252f1a5e","Type":"ContainerDied","Data":"ad3fc9833472364fa84789f9678f9810a51013259631796bd31252854a2725ad"} Jul 06 15:55:23.330764 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:23.330696 2561 scope.go:117] "RemoveContainer" containerID="ac8b8f28ae230cd8e6ce6692d8cfa9bbdc9ae7fb167f99e70b0671117f5c6163" Jul 06 15:55:23.356468 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:23.356442 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks"] Jul 06 15:55:23.364121 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:23.364098 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-47b69c3c-7c9d84bf66-skhks"] Jul 06 15:55:24.801799 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:24.801771 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f4918e0-a836-4f64-85ec-0bb6252f1a5e" path="/var/lib/kubelet/pods/6f4918e0-a836-4f64-85ec-0bb6252f1a5e/volumes" Jul 06 15:55:27.835224 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:27.835186 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx"] Jul 06 15:55:27.835652 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:27.835424 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" podUID="931bed4e-bb0c-45d9-8c0e-12b102c53f3d" containerName="istio-proxy" containerID="cri-o://e2b013208ec0d9b059dda1db81d0bef25a5a74662f5ae9b416e39607c063d96a" gracePeriod=30 Jul 06 15:55:27.876306 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:27.875254 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:27.981448 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:27.981417 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:28.199634 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.199611 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:28.234261 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.234233 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-envoy\") pod \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " Jul 06 15:55:28.234404 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.234269 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-workload-certs\") pod \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " Jul 06 15:55:28.234404 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.234300 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-workload-socket\") pod \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " Jul 06 15:55:28.234404 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.234348 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-credential-socket\") pod \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " Jul 06 15:55:28.234599 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.234413 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-token\") pod \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " Jul 06 15:55:28.234599 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.234458 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-data\") pod \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " Jul 06 15:55:28.234599 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.234491 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istiod-ca-cert\") pod \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " Jul 06 15:55:28.234599 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.234545 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwwcs\" (UniqueName: \"kubernetes.io/projected/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-kube-api-access-gwwcs\") pod \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " Jul 06 15:55:28.234599 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.234569 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "931bed4e-bb0c-45d9-8c0e-12b102c53f3d" (UID: "931bed4e-bb0c-45d9-8c0e-12b102c53f3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:28.234599 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.234595 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-podinfo\") pod \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\" (UID: \"931bed4e-bb0c-45d9-8c0e-12b102c53f3d\") " Jul 06 15:55:28.234885 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.234732 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "931bed4e-bb0c-45d9-8c0e-12b102c53f3d" (UID: "931bed4e-bb0c-45d9-8c0e-12b102c53f3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:28.234942 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.234892 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "931bed4e-bb0c-45d9-8c0e-12b102c53f3d" (UID: "931bed4e-bb0c-45d9-8c0e-12b102c53f3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:28.235044 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.235024 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-workload-certs\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:28.235103 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.235052 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-workload-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:28.235103 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.235066 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-credential-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:28.235202 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.235170 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "931bed4e-bb0c-45d9-8c0e-12b102c53f3d" (UID: "931bed4e-bb0c-45d9-8c0e-12b102c53f3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:55:28.236026 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.235988 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-data" (OuterVolumeSpecName: "istio-data") pod "931bed4e-bb0c-45d9-8c0e-12b102c53f3d" (UID: "931bed4e-bb0c-45d9-8c0e-12b102c53f3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:28.238336 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.238218 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-kube-api-access-gwwcs" (OuterVolumeSpecName: "kube-api-access-gwwcs") pod "931bed4e-bb0c-45d9-8c0e-12b102c53f3d" (UID: "931bed4e-bb0c-45d9-8c0e-12b102c53f3d"). InnerVolumeSpecName "kube-api-access-gwwcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:55:28.238473 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.238280 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "931bed4e-bb0c-45d9-8c0e-12b102c53f3d" (UID: "931bed4e-bb0c-45d9-8c0e-12b102c53f3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:28.238473 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.238288 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "931bed4e-bb0c-45d9-8c0e-12b102c53f3d" (UID: "931bed4e-bb0c-45d9-8c0e-12b102c53f3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 15:55:28.238650 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.238489 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-token" (OuterVolumeSpecName: "istio-token") pod "931bed4e-bb0c-45d9-8c0e-12b102c53f3d" (UID: "931bed4e-bb0c-45d9-8c0e-12b102c53f3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:55:28.335645 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.335618 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-envoy\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:28.335645 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.335645 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-token\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:28.335811 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.335656 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-data\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:28.335811 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.335663 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istiod-ca-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:28.335811 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.335671 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwwcs\" (UniqueName: \"kubernetes.io/projected/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-kube-api-access-gwwcs\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:28.335811 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.335679 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/931bed4e-bb0c-45d9-8c0e-12b102c53f3d-istio-podinfo\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:28.362753 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.362682 2561 generic.go:356] "Generic (PLEG): container finished" podID="931bed4e-bb0c-45d9-8c0e-12b102c53f3d" containerID="e2b013208ec0d9b059dda1db81d0bef25a5a74662f5ae9b416e39607c063d96a" exitCode=0 Jul 06 15:55:28.362753 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.362733 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" Jul 06 15:55:28.362753 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.362747 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" event={"ID":"931bed4e-bb0c-45d9-8c0e-12b102c53f3d","Type":"ContainerDied","Data":"e2b013208ec0d9b059dda1db81d0bef25a5a74662f5ae9b416e39607c063d96a"} Jul 06 15:55:28.362950 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.362792 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx" event={"ID":"931bed4e-bb0c-45d9-8c0e-12b102c53f3d","Type":"ContainerDied","Data":"759678a9bd2405e91dbae54af67a778a2754d94f8ba2b462715e35c9ca3c1416"} Jul 06 15:55:28.362950 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.362812 2561 scope.go:117] "RemoveContainer" containerID="e2b013208ec0d9b059dda1db81d0bef25a5a74662f5ae9b416e39607c063d96a" Jul 06 15:55:28.372095 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.372079 2561 scope.go:117] "RemoveContainer" containerID="e2b013208ec0d9b059dda1db81d0bef25a5a74662f5ae9b416e39607c063d96a" Jul 06 15:55:28.372314 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:55:28.372298 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b013208ec0d9b059dda1db81d0bef25a5a74662f5ae9b416e39607c063d96a\": container with ID starting with e2b013208ec0d9b059dda1db81d0bef25a5a74662f5ae9b416e39607c063d96a not found: ID does not exist" containerID="e2b013208ec0d9b059dda1db81d0bef25a5a74662f5ae9b416e39607c063d96a" Jul 06 15:55:28.372354 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.372321 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b013208ec0d9b059dda1db81d0bef25a5a74662f5ae9b416e39607c063d96a"} err="failed to get container status \"e2b013208ec0d9b059dda1db81d0bef25a5a74662f5ae9b416e39607c063d96a\": rpc error: code = NotFound desc = could not find container \"e2b013208ec0d9b059dda1db81d0bef25a5a74662f5ae9b416e39607c063d96a\": container with ID starting with e2b013208ec0d9b059dda1db81d0bef25a5a74662f5ae9b416e39607c063d96a not found: ID does not exist" Jul 06 15:55:28.391482 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.391462 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx"] Jul 06 15:55:28.395549 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.395514 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-47b69c3c-openshift-default-74bdc486df-zrtmx"] Jul 06 15:55:28.800467 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:28.800426 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="931bed4e-bb0c-45d9-8c0e-12b102c53f3d" path="/var/lib/kubelet/pods/931bed4e-bb0c-45d9-8c0e-12b102c53f3d/volumes" Jul 06 15:55:29.311812 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:29.311779 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:34.522896 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.522842 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:34.630666 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.630632 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:34.730980 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.730952 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd"] Jul 06 15:55:34.731414 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.731398 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="931bed4e-bb0c-45d9-8c0e-12b102c53f3d" containerName="istio-proxy" Jul 06 15:55:34.731414 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.731411 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="931bed4e-bb0c-45d9-8c0e-12b102c53f3d" containerName="istio-proxy" Jul 06 15:55:34.731596 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.731439 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4918e0-a836-4f64-85ec-0bb6252f1a5e" containerName="maas-api" Jul 06 15:55:34.731596 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.731445 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4918e0-a836-4f64-85ec-0bb6252f1a5e" containerName="maas-api" Jul 06 15:55:34.731596 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.731507 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="931bed4e-bb0c-45d9-8c0e-12b102c53f3d" containerName="istio-proxy" Jul 06 15:55:34.731596 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.731519 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4918e0-a836-4f64-85ec-0bb6252f1a5e" containerName="maas-api" Jul 06 15:55:34.736422 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.736401 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.739557 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.739516 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-69284b57-openshift-default-dockercfg-d6h79\"" Jul 06 15:55:34.754117 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.754097 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd"] Jul 06 15:55:34.794023 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.793958 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-podinfo\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.794023 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.794006 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-token\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.794180 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.794125 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgz85\" (UniqueName: \"kubernetes.io/projected/227270c1-cff0-47db-b8d3-1f0e92a67d89-kube-api-access-cgz85\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.794301 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.794277 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-data\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.794363 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.794346 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-workload-certs\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.794442 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.794424 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-credential-socket\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.794485 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.794476 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-envoy\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.794577 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.794558 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227270c1-cff0-47db-b8d3-1f0e92a67d89-istiod-ca-cert\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.794632 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.794601 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-workload-socket\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.804435 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.804414 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:34.895828 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.895793 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-credential-socket\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.896017 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.895836 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-envoy\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.896017 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.895869 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227270c1-cff0-47db-b8d3-1f0e92a67d89-istiod-ca-cert\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.896017 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.895892 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-workload-socket\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.896017 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.895958 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-podinfo\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.896017 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.896005 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-token\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.896281 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.896029 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgz85\" (UniqueName: \"kubernetes.io/projected/227270c1-cff0-47db-b8d3-1f0e92a67d89-kube-api-access-cgz85\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.896281 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.896084 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-data\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.896281 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.896164 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-workload-certs\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.896281 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.896203 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-credential-socket\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.896586 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.896303 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-workload-socket\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.896586 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.896470 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-data\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.896586 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.896573 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-workload-certs\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.896736 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.896686 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227270c1-cff0-47db-b8d3-1f0e92a67d89-istiod-ca-cert\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.898330 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.898307 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-envoy\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.898783 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.898757 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:34.898987 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.898972 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-podinfo\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.906002 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.905981 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-token\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:34.906099 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:34.906081 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgz85\" (UniqueName: \"kubernetes.io/projected/227270c1-cff0-47db-b8d3-1f0e92a67d89-kube-api-access-cgz85\") pod \"e2e-derive-69284b57-openshift-default-8677cdb575-z49zd\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:35.046589 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:35.046476 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:35.218582 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:55:35.218347 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227270c1_cff0_47db_b8d3_1f0e92a67d89.slice/crio-e39e9aec57b21a96532e351d39354b4e02ca61bf445be37b508845274af3dba2 WatchSource:0}: Error finding container e39e9aec57b21a96532e351d39354b4e02ca61bf445be37b508845274af3dba2: Status 404 returned error can't find the container with id e39e9aec57b21a96532e351d39354b4e02ca61bf445be37b508845274af3dba2 Jul 06 15:55:35.220685 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:35.220653 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd"] Jul 06 15:55:35.220976 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:35.220946 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:55:35.221026 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:35.221002 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:55:35.221063 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:35.221029 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:55:35.408371 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:35.407620 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" event={"ID":"227270c1-cff0-47db-b8d3-1f0e92a67d89","Type":"ContainerStarted","Data":"492f5b6a0505bd4acc3da3eb5139ae65704a72dd1cfb2edade0a76a53c44f67c"} Jul 06 15:55:35.408371 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:35.407714 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" event={"ID":"227270c1-cff0-47db-b8d3-1f0e92a67d89","Type":"ContainerStarted","Data":"e39e9aec57b21a96532e351d39354b4e02ca61bf445be37b508845274af3dba2"} Jul 06 15:55:35.430422 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:35.430375 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" podStartSLOduration=1.430360911 podStartE2EDuration="1.430360911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:55:35.429083075 +0000 UTC m=+1757.061635061" watchObservedRunningTime="2026-07-06 15:55:35.430360911 +0000 UTC m=+1757.062912894" Jul 06 15:55:35.963507 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:35.963454 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:36.047580 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.047518 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:36.052373 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.052353 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:36.058389 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.058368 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:36.178834 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.178804 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg"] Jul 06 15:55:36.182654 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.182632 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" Jul 06 15:55:36.185471 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.185447 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-69284b57-serving-cert\"" Jul 06 15:55:36.185611 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.185569 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rszzk\"" Jul 06 15:55:36.194081 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.194059 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg"] Jul 06 15:55:36.241921 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.241895 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:36.314908 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.314877 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/802040cc-7c5c-44bb-b948-ad788a108d57-maas-api-tls\") pod \"maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg\" (UID: \"802040cc-7c5c-44bb-b948-ad788a108d57\") " pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" Jul 06 15:55:36.315033 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.314927 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cljv\" (UniqueName: \"kubernetes.io/projected/802040cc-7c5c-44bb-b948-ad788a108d57-kube-api-access-6cljv\") pod \"maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg\" (UID: \"802040cc-7c5c-44bb-b948-ad788a108d57\") " pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" Jul 06 15:55:36.315080 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.315044 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/802040cc-7c5c-44bb-b948-ad788a108d57-openshift-service-ca\") pod \"maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg\" (UID: \"802040cc-7c5c-44bb-b948-ad788a108d57\") " pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" Jul 06 15:55:36.413347 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.413316 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:36.414218 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.414198 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:36.415583 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.415561 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cljv\" (UniqueName: \"kubernetes.io/projected/802040cc-7c5c-44bb-b948-ad788a108d57-kube-api-access-6cljv\") pod \"maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg\" (UID: \"802040cc-7c5c-44bb-b948-ad788a108d57\") " pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" Jul 06 15:55:36.415644 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.415629 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/802040cc-7c5c-44bb-b948-ad788a108d57-openshift-service-ca\") pod \"maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg\" (UID: \"802040cc-7c5c-44bb-b948-ad788a108d57\") " pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" Jul 06 15:55:36.415706 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.415698 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/802040cc-7c5c-44bb-b948-ad788a108d57-maas-api-tls\") pod \"maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg\" (UID: \"802040cc-7c5c-44bb-b948-ad788a108d57\") " pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" Jul 06 15:55:36.416239 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.416220 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/802040cc-7c5c-44bb-b948-ad788a108d57-openshift-service-ca\") pod \"maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg\" (UID: \"802040cc-7c5c-44bb-b948-ad788a108d57\") " pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" Jul 06 15:55:36.418282 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.418253 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/802040cc-7c5c-44bb-b948-ad788a108d57-maas-api-tls\") pod \"maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg\" (UID: \"802040cc-7c5c-44bb-b948-ad788a108d57\") " pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" Jul 06 15:55:36.420283 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.420266 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:36.425718 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.425696 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cljv\" (UniqueName: \"kubernetes.io/projected/802040cc-7c5c-44bb-b948-ad788a108d57-kube-api-access-6cljv\") pod \"maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg\" (UID: \"802040cc-7c5c-44bb-b948-ad788a108d57\") " pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" Jul 06 15:55:36.496744 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.496668 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" Jul 06 15:55:36.645491 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.645465 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg"] Jul 06 15:55:36.646180 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:55:36.646158 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802040cc_7c5c_44bb_b948_ad788a108d57.slice/crio-8674cebe26d1a5306b9694e1fbd5e8131ef13ca422a4c572c88272a023668887 WatchSource:0}: Error finding container 8674cebe26d1a5306b9694e1fbd5e8131ef13ca422a4c572c88272a023668887: Status 404 returned error can't find the container with id 8674cebe26d1a5306b9694e1fbd5e8131ef13ca422a4c572c88272a023668887 Jul 06 15:55:36.919797 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:36.919720 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:37.016049 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:37.016014 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:37.420283 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:37.420248 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" event={"ID":"802040cc-7c5c-44bb-b948-ad788a108d57","Type":"ContainerStarted","Data":"4e36a07b747e44663f70313fce363a217eca8bbfe9324cf778b236aeeae0cadb"} Jul 06 15:55:37.420465 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:37.420310 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" event={"ID":"802040cc-7c5c-44bb-b948-ad788a108d57","Type":"ContainerStarted","Data":"8674cebe26d1a5306b9694e1fbd5e8131ef13ca422a4c572c88272a023668887"} Jul 06 15:55:37.420465 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:37.420442 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" Jul 06 15:55:37.446375 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:37.446332 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" podStartSLOduration=1.074455441 podStartE2EDuration="1.446317382s" totalImagesPullingTime="371.861941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:55:37.444092776 +0000 UTC m=+1759.076644758" watchObservedRunningTime="2026-07-06 15:55:37.446317382 +0000 UTC m=+1759.078869365" Jul 06 15:55:38.393884 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:38.393858 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:38.914720 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:38.914686 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:40.481556 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:40.481488 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:40.620325 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:40.620288 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd"] Jul 06 15:55:40.621058 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:40.621025 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" podUID="227270c1-cff0-47db-b8d3-1f0e92a67d89" containerName="istio-proxy" containerID="cri-o://492f5b6a0505bd4acc3da3eb5139ae65704a72dd1cfb2edade0a76a53c44f67c" gracePeriod=30 Jul 06 15:55:40.666160 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:40.666132 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:40.744392 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:40.744294 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:40.953900 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:40.953870 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg"] Jul 06 15:55:40.957715 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:40.954302 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" podUID="802040cc-7c5c-44bb-b948-ad788a108d57" containerName="maas-api" containerID="cri-o://4e36a07b747e44663f70313fce363a217eca8bbfe9324cf778b236aeeae0cadb" gracePeriod=30 Jul 06 15:55:40.985689 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:40.985661 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:41.269414 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.269394 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" Jul 06 15:55:41.367411 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.367332 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/802040cc-7c5c-44bb-b948-ad788a108d57-maas-api-tls\") pod \"802040cc-7c5c-44bb-b948-ad788a108d57\" (UID: \"802040cc-7c5c-44bb-b948-ad788a108d57\") " Jul 06 15:55:41.367411 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.367399 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cljv\" (UniqueName: \"kubernetes.io/projected/802040cc-7c5c-44bb-b948-ad788a108d57-kube-api-access-6cljv\") pod \"802040cc-7c5c-44bb-b948-ad788a108d57\" (UID: \"802040cc-7c5c-44bb-b948-ad788a108d57\") " Jul 06 15:55:41.367635 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.367438 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/802040cc-7c5c-44bb-b948-ad788a108d57-openshift-service-ca\") pod \"802040cc-7c5c-44bb-b948-ad788a108d57\" (UID: \"802040cc-7c5c-44bb-b948-ad788a108d57\") " Jul 06 15:55:41.367907 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.367873 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802040cc-7c5c-44bb-b948-ad788a108d57-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "802040cc-7c5c-44bb-b948-ad788a108d57" (UID: "802040cc-7c5c-44bb-b948-ad788a108d57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:55:41.369587 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.369565 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/802040cc-7c5c-44bb-b948-ad788a108d57-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "802040cc-7c5c-44bb-b948-ad788a108d57" (UID: "802040cc-7c5c-44bb-b948-ad788a108d57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:55:41.369761 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.369739 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/802040cc-7c5c-44bb-b948-ad788a108d57-kube-api-access-6cljv" (OuterVolumeSpecName: "kube-api-access-6cljv") pod "802040cc-7c5c-44bb-b948-ad788a108d57" (UID: "802040cc-7c5c-44bb-b948-ad788a108d57"). InnerVolumeSpecName "kube-api-access-6cljv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:55:41.448290 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.448256 2561 generic.go:356] "Generic (PLEG): container finished" podID="802040cc-7c5c-44bb-b948-ad788a108d57" containerID="4e36a07b747e44663f70313fce363a217eca8bbfe9324cf778b236aeeae0cadb" exitCode=0 Jul 06 15:55:41.448445 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.448296 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" Jul 06 15:55:41.448445 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.448337 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" event={"ID":"802040cc-7c5c-44bb-b948-ad788a108d57","Type":"ContainerDied","Data":"4e36a07b747e44663f70313fce363a217eca8bbfe9324cf778b236aeeae0cadb"} Jul 06 15:55:41.448445 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.448387 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg" event={"ID":"802040cc-7c5c-44bb-b948-ad788a108d57","Type":"ContainerDied","Data":"8674cebe26d1a5306b9694e1fbd5e8131ef13ca422a4c572c88272a023668887"} Jul 06 15:55:41.448445 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.448413 2561 scope.go:117] "RemoveContainer" containerID="4e36a07b747e44663f70313fce363a217eca8bbfe9324cf778b236aeeae0cadb" Jul 06 15:55:41.459052 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.459031 2561 scope.go:117] "RemoveContainer" containerID="4e36a07b747e44663f70313fce363a217eca8bbfe9324cf778b236aeeae0cadb" Jul 06 15:55:41.459329 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:55:41.459310 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e36a07b747e44663f70313fce363a217eca8bbfe9324cf778b236aeeae0cadb\": container with ID starting with 4e36a07b747e44663f70313fce363a217eca8bbfe9324cf778b236aeeae0cadb not found: ID does not exist" containerID="4e36a07b747e44663f70313fce363a217eca8bbfe9324cf778b236aeeae0cadb" Jul 06 15:55:41.459386 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.459340 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e36a07b747e44663f70313fce363a217eca8bbfe9324cf778b236aeeae0cadb"} err="failed to get container status \"4e36a07b747e44663f70313fce363a217eca8bbfe9324cf778b236aeeae0cadb\": rpc error: code = NotFound desc = could not find container \"4e36a07b747e44663f70313fce363a217eca8bbfe9324cf778b236aeeae0cadb\": container with ID starting with 4e36a07b747e44663f70313fce363a217eca8bbfe9324cf778b236aeeae0cadb not found: ID does not exist" Jul 06 15:55:41.468567 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.468550 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/802040cc-7c5c-44bb-b948-ad788a108d57-maas-api-tls\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:41.468649 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.468570 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cljv\" (UniqueName: \"kubernetes.io/projected/802040cc-7c5c-44bb-b948-ad788a108d57-kube-api-access-6cljv\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:41.468649 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.468579 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/802040cc-7c5c-44bb-b948-ad788a108d57-openshift-service-ca\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:41.486338 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.486317 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg"] Jul 06 15:55:41.489117 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:41.489096 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-69284b57-5d88699d9c-xfnfg"] Jul 06 15:55:42.086574 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:42.086519 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:42.217716 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:42.217676 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:42.553722 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:42.553696 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:42.800293 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:42.800208 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="802040cc-7c5c-44bb-b948-ad788a108d57" path="/var/lib/kubelet/pods/802040cc-7c5c-44bb-b948-ad788a108d57/volumes" Jul 06 15:55:47.391761 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.391738 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:47.474979 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.474951 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:47.483680 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.483653 2561 generic.go:356] "Generic (PLEG): container finished" podID="227270c1-cff0-47db-b8d3-1f0e92a67d89" containerID="492f5b6a0505bd4acc3da3eb5139ae65704a72dd1cfb2edade0a76a53c44f67c" exitCode=0 Jul 06 15:55:47.483835 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.483699 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" Jul 06 15:55:47.483835 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.483735 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" event={"ID":"227270c1-cff0-47db-b8d3-1f0e92a67d89","Type":"ContainerDied","Data":"492f5b6a0505bd4acc3da3eb5139ae65704a72dd1cfb2edade0a76a53c44f67c"} Jul 06 15:55:47.483835 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.483797 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd" event={"ID":"227270c1-cff0-47db-b8d3-1f0e92a67d89","Type":"ContainerDied","Data":"e39e9aec57b21a96532e351d39354b4e02ca61bf445be37b508845274af3dba2"} Jul 06 15:55:47.483835 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.483828 2561 scope.go:117] "RemoveContainer" containerID="492f5b6a0505bd4acc3da3eb5139ae65704a72dd1cfb2edade0a76a53c44f67c" Jul 06 15:55:47.497167 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.497148 2561 scope.go:117] "RemoveContainer" containerID="492f5b6a0505bd4acc3da3eb5139ae65704a72dd1cfb2edade0a76a53c44f67c" Jul 06 15:55:47.497412 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:55:47.497396 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492f5b6a0505bd4acc3da3eb5139ae65704a72dd1cfb2edade0a76a53c44f67c\": container with ID starting with 492f5b6a0505bd4acc3da3eb5139ae65704a72dd1cfb2edade0a76a53c44f67c not found: ID does not exist" containerID="492f5b6a0505bd4acc3da3eb5139ae65704a72dd1cfb2edade0a76a53c44f67c" Jul 06 15:55:47.497467 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.497429 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492f5b6a0505bd4acc3da3eb5139ae65704a72dd1cfb2edade0a76a53c44f67c"} err="failed to get container status \"492f5b6a0505bd4acc3da3eb5139ae65704a72dd1cfb2edade0a76a53c44f67c\": rpc error: code = NotFound desc = could not find container \"492f5b6a0505bd4acc3da3eb5139ae65704a72dd1cfb2edade0a76a53c44f67c\": container with ID starting with 492f5b6a0505bd4acc3da3eb5139ae65704a72dd1cfb2edade0a76a53c44f67c not found: ID does not exist" Jul 06 15:55:47.532817 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.532794 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-podinfo\") pod \"227270c1-cff0-47db-b8d3-1f0e92a67d89\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " Jul 06 15:55:47.532959 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.532830 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgz85\" (UniqueName: \"kubernetes.io/projected/227270c1-cff0-47db-b8d3-1f0e92a67d89-kube-api-access-cgz85\") pod \"227270c1-cff0-47db-b8d3-1f0e92a67d89\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " Jul 06 15:55:47.532959 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.532861 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227270c1-cff0-47db-b8d3-1f0e92a67d89-istiod-ca-cert\") pod \"227270c1-cff0-47db-b8d3-1f0e92a67d89\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " Jul 06 15:55:47.532959 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.532891 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-workload-certs\") pod \"227270c1-cff0-47db-b8d3-1f0e92a67d89\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " Jul 06 15:55:47.532959 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.532952 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-token\") pod \"227270c1-cff0-47db-b8d3-1f0e92a67d89\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " Jul 06 15:55:47.533146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.532990 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-workload-socket\") pod \"227270c1-cff0-47db-b8d3-1f0e92a67d89\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " Jul 06 15:55:47.533199 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.533160 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-credential-socket\") pod \"227270c1-cff0-47db-b8d3-1f0e92a67d89\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " Jul 06 15:55:47.533258 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.533246 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "227270c1-cff0-47db-b8d3-1f0e92a67d89" (UID: "227270c1-cff0-47db-b8d3-1f0e92a67d89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:47.533258 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.533254 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-data\") pod \"227270c1-cff0-47db-b8d3-1f0e92a67d89\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " Jul 06 15:55:47.533366 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.533324 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-envoy\") pod \"227270c1-cff0-47db-b8d3-1f0e92a67d89\" (UID: \"227270c1-cff0-47db-b8d3-1f0e92a67d89\") " Jul 06 15:55:47.533430 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.533373 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "227270c1-cff0-47db-b8d3-1f0e92a67d89" (UID: "227270c1-cff0-47db-b8d3-1f0e92a67d89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:47.533614 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.533448 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "227270c1-cff0-47db-b8d3-1f0e92a67d89" (UID: "227270c1-cff0-47db-b8d3-1f0e92a67d89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:47.533697 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.533517 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227270c1-cff0-47db-b8d3-1f0e92a67d89-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "227270c1-cff0-47db-b8d3-1f0e92a67d89" (UID: "227270c1-cff0-47db-b8d3-1f0e92a67d89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:55:47.533804 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.533789 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/227270c1-cff0-47db-b8d3-1f0e92a67d89-istiod-ca-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:47.533952 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.533939 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-workload-certs\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:47.534341 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.534300 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-workload-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:47.534341 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.534320 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-credential-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:47.534608 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.534552 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-data" (OuterVolumeSpecName: "istio-data") pod "227270c1-cff0-47db-b8d3-1f0e92a67d89" (UID: "227270c1-cff0-47db-b8d3-1f0e92a67d89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:47.535397 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.535360 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "227270c1-cff0-47db-b8d3-1f0e92a67d89" (UID: "227270c1-cff0-47db-b8d3-1f0e92a67d89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 15:55:47.535876 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.535853 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227270c1-cff0-47db-b8d3-1f0e92a67d89-kube-api-access-cgz85" (OuterVolumeSpecName: "kube-api-access-cgz85") pod "227270c1-cff0-47db-b8d3-1f0e92a67d89" (UID: "227270c1-cff0-47db-b8d3-1f0e92a67d89"). InnerVolumeSpecName "kube-api-access-cgz85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:55:47.536005 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.535911 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "227270c1-cff0-47db-b8d3-1f0e92a67d89" (UID: "227270c1-cff0-47db-b8d3-1f0e92a67d89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:55:47.536072 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.536055 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-token" (OuterVolumeSpecName: "istio-token") pod "227270c1-cff0-47db-b8d3-1f0e92a67d89" (UID: "227270c1-cff0-47db-b8d3-1f0e92a67d89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:55:47.574956 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.574930 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:47.635474 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.635441 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-token\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:47.635474 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.635472 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-data\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:47.635670 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.635484 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-envoy\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:47.635670 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.635497 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/227270c1-cff0-47db-b8d3-1f0e92a67d89-istio-podinfo\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:47.635670 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.635509 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgz85\" (UniqueName: \"kubernetes.io/projected/227270c1-cff0-47db-b8d3-1f0e92a67d89-kube-api-access-cgz85\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:55:47.675622 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.675595 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:47.819429 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.819397 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd"] Jul 06 15:55:47.830348 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:47.830323 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-69284b57-openshift-default-8677cdb575-z49zd"] Jul 06 15:55:48.802148 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:48.802119 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="227270c1-cff0-47db-b8d3-1f0e92a67d89" path="/var/lib/kubelet/pods/227270c1-cff0-47db-b8d3-1f0e92a67d89/volumes" Jul 06 15:55:49.239217 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:49.239189 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:54.991442 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:54.991410 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:56.197315 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:56.197286 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:56.301098 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:56.301066 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:56.406435 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:56.406397 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:56.524319 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:56.524293 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:56.932500 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:55:56.932381 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:18.871651 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:56:18.871622 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 15:56:18.874486 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:56:18.874462 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 15:56:22.926416 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:56:22.926379 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:29.376540 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:56:29.376496 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:29.495179 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:56:29.495139 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:29.609436 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:56:29.609405 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:45.556456 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:56:45.556419 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:51.914164 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:56:51.914133 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:52.024705 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:56:52.024662 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:52.117628 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:56:52.117594 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:02.648747 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:02.648713 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:03.472189 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:03.472152 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:09.856830 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:09.856792 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:09.962988 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:09.962954 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:10.105700 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:10.105665 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:10.206192 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:10.206112 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:11.012000 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:11.011969 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:11.108606 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:11.108563 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:11.179717 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:11.179692 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:12.888144 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:12.888116 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:18.665753 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:18.665715 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:18.763651 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:18.763620 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:19.610790 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:19.610755 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:19.711324 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:19.711287 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:20.321662 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:20.321623 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:27.479430 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:27.479394 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:27.578332 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:27.578302 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:27.729868 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:27.729791 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:28.057056 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.056978 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd"] Jul 06 15:57:28.057457 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.057440 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="227270c1-cff0-47db-b8d3-1f0e92a67d89" containerName="istio-proxy" Jul 06 15:57:28.057575 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.057460 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="227270c1-cff0-47db-b8d3-1f0e92a67d89" containerName="istio-proxy" Jul 06 15:57:28.057575 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.057492 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="802040cc-7c5c-44bb-b948-ad788a108d57" containerName="maas-api" Jul 06 15:57:28.057575 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.057501 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="802040cc-7c5c-44bb-b948-ad788a108d57" containerName="maas-api" Jul 06 15:57:28.057747 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.057634 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="802040cc-7c5c-44bb-b948-ad788a108d57" containerName="maas-api" Jul 06 15:57:28.057747 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.057649 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="227270c1-cff0-47db-b8d3-1f0e92a67d89" containerName="istio-proxy" Jul 06 15:57:28.060617 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.060587 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.064075 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.064050 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-37904f1b-openshift-default-dockercfg-t7pf4\"" Jul 06 15:57:28.095352 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.095330 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd"] Jul 06 15:57:28.102349 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.102319 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:28.118314 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.118284 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-workload-socket\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.118415 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.118352 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-podinfo\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.118415 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.118393 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-envoy\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.118497 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.118424 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-token\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.118497 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.118480 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istiod-ca-cert\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.118605 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.118512 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-credential-socket\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.118605 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.118577 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-workload-certs\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.118681 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.118617 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-data\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.118718 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.118666 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4nnw\" (UniqueName: \"kubernetes.io/projected/ee3f6509-7558-4303-8eda-9fbd4f8400c3-kube-api-access-c4nnw\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.219245 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.219212 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-token\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.219412 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.219255 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istiod-ca-cert\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.219412 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.219273 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-credential-socket\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.219412 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.219295 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-workload-certs\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.219412 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.219334 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-data\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.219412 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.219361 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4nnw\" (UniqueName: \"kubernetes.io/projected/ee3f6509-7558-4303-8eda-9fbd4f8400c3-kube-api-access-c4nnw\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.219709 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.219416 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-workload-socket\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.219709 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.219502 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-podinfo\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.219709 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.219587 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-envoy\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.219864 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.219775 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-credential-socket\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.219864 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.219802 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-workload-certs\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.219994 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.219969 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-workload-socket\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.220222 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.220194 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istiod-ca-cert\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.220519 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.220276 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-data\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.222222 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.222196 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-podinfo\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.222318 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.222241 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-envoy\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.231669 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.231646 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-token\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.232021 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.231992 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4nnw\" (UniqueName: \"kubernetes.io/projected/ee3f6509-7558-4303-8eda-9fbd4f8400c3-kube-api-access-c4nnw\") pod \"e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.371126 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.371046 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:28.528880 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.528851 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd"] Jul 06 15:57:28.530696 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:57:28.530668 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3f6509_7558_4303_8eda_9fbd4f8400c3.slice/crio-e68ccb2b48992fc7224f46c7da7449180f6be61c2250a454f35a882c685adcd4 WatchSource:0}: Error finding container e68ccb2b48992fc7224f46c7da7449180f6be61c2250a454f35a882c685adcd4: Status 404 returned error can't find the container with id e68ccb2b48992fc7224f46c7da7449180f6be61c2250a454f35a882c685adcd4 Jul 06 15:57:28.532978 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.532943 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:57:28.533061 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.533014 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:57:28.533061 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.533043 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:57:28.667187 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:28.667096 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:29.009125 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:29.009077 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" event={"ID":"ee3f6509-7558-4303-8eda-9fbd4f8400c3","Type":"ContainerStarted","Data":"9c590384ec2094cd415ecb69a731e759d10dce9bfc63d5e28a75c3ab6b487baa"} Jul 06 15:57:29.009305 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:29.009139 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" event={"ID":"ee3f6509-7558-4303-8eda-9fbd4f8400c3","Type":"ContainerStarted","Data":"e68ccb2b48992fc7224f46c7da7449180f6be61c2250a454f35a882c685adcd4"} Jul 06 15:57:29.045292 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:29.045239 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" podStartSLOduration=1.045225404 podStartE2EDuration="1.045225404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:57:29.043965908 +0000 UTC m=+1870.676517892" watchObservedRunningTime="2026-07-06 15:57:29.045225404 +0000 UTC m=+1870.677777386" Jul 06 15:57:29.371285 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:29.371210 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:29.376397 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:29.376365 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:30.016133 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:30.016098 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:30.017136 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:30.017118 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:31.141827 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.141792 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:31.210635 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.210596 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd"] Jul 06 15:57:31.214702 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.214680 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:31.217696 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.217671 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-37904f1b-serving-cert\"" Jul 06 15:57:31.217842 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.217698 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rszzk\"" Jul 06 15:57:31.227451 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.227428 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd"] Jul 06 15:57:31.231992 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.231968 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:31.245790 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.245763 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2df0eb-9830-477b-9623-2c2763db601a-openshift-service-ca\") pod \"maas-api-e2e-mt-37904f1b-78d475cc86-t94sd\" (UID: \"ee2df0eb-9830-477b-9623-2c2763db601a\") " pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:31.245907 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.245797 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khk2m\" (UniqueName: \"kubernetes.io/projected/ee2df0eb-9830-477b-9623-2c2763db601a-kube-api-access-khk2m\") pod \"maas-api-e2e-mt-37904f1b-78d475cc86-t94sd\" (UID: \"ee2df0eb-9830-477b-9623-2c2763db601a\") " pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:31.245970 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.245943 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee2df0eb-9830-477b-9623-2c2763db601a-maas-api-tls\") pod \"maas-api-e2e-mt-37904f1b-78d475cc86-t94sd\" (UID: \"ee2df0eb-9830-477b-9623-2c2763db601a\") " pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:31.311190 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.311143 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:31.347492 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.347459 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee2df0eb-9830-477b-9623-2c2763db601a-maas-api-tls\") pod \"maas-api-e2e-mt-37904f1b-78d475cc86-t94sd\" (UID: \"ee2df0eb-9830-477b-9623-2c2763db601a\") " pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:31.347666 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.347556 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2df0eb-9830-477b-9623-2c2763db601a-openshift-service-ca\") pod \"maas-api-e2e-mt-37904f1b-78d475cc86-t94sd\" (UID: \"ee2df0eb-9830-477b-9623-2c2763db601a\") " pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:31.347666 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.347587 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khk2m\" (UniqueName: \"kubernetes.io/projected/ee2df0eb-9830-477b-9623-2c2763db601a-kube-api-access-khk2m\") pod \"maas-api-e2e-mt-37904f1b-78d475cc86-t94sd\" (UID: \"ee2df0eb-9830-477b-9623-2c2763db601a\") " pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:31.347666 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:57:31.347633 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-37904f1b-serving-cert: secret "maas-api-e2e-mt-37904f1b-serving-cert" not found Jul 06 15:57:31.347847 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:57:31.347695 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee2df0eb-9830-477b-9623-2c2763db601a-maas-api-tls podName:ee2df0eb-9830-477b-9623-2c2763db601a nodeName:}" failed. No retries permitted until 2026-07-06 15:57:31.847674482 +0000 UTC m=+1873.480226444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee2df0eb-9830-477b-9623-2c2763db601a-maas-api-tls") pod "maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" (UID: "ee2df0eb-9830-477b-9623-2c2763db601a") : secret "maas-api-e2e-mt-37904f1b-serving-cert" not found Jul 06 15:57:31.348176 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.348158 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2df0eb-9830-477b-9623-2c2763db601a-openshift-service-ca\") pod \"maas-api-e2e-mt-37904f1b-78d475cc86-t94sd\" (UID: \"ee2df0eb-9830-477b-9623-2c2763db601a\") " pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:31.360954 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.360931 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khk2m\" (UniqueName: \"kubernetes.io/projected/ee2df0eb-9830-477b-9623-2c2763db601a-kube-api-access-khk2m\") pod \"maas-api-e2e-mt-37904f1b-78d475cc86-t94sd\" (UID: \"ee2df0eb-9830-477b-9623-2c2763db601a\") " pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:31.512422 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.512396 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:31.763578 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.763418 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:31.852512 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.852481 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee2df0eb-9830-477b-9623-2c2763db601a-maas-api-tls\") pod \"maas-api-e2e-mt-37904f1b-78d475cc86-t94sd\" (UID: \"ee2df0eb-9830-477b-9623-2c2763db601a\") " pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:31.855074 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.855047 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee2df0eb-9830-477b-9623-2c2763db601a-maas-api-tls\") pod \"maas-api-e2e-mt-37904f1b-78d475cc86-t94sd\" (UID: \"ee2df0eb-9830-477b-9623-2c2763db601a\") " pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:31.865284 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:31.865243 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:32.129325 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:32.129229 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:32.290217 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:32.290195 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd"] Jul 06 15:57:32.292092 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:57:32.292060 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee2df0eb_9830_477b_9623_2c2763db601a.slice/crio-5be7d49f91cc208f8f0b6cbc8db5da3984c37f8f74869cf15091db53dfcbeef6 WatchSource:0}: Error finding container 5be7d49f91cc208f8f0b6cbc8db5da3984c37f8f74869cf15091db53dfcbeef6: Status 404 returned error can't find the container with id 5be7d49f91cc208f8f0b6cbc8db5da3984c37f8f74869cf15091db53dfcbeef6 Jul 06 15:57:33.035560 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:33.035497 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" event={"ID":"ee2df0eb-9830-477b-9623-2c2763db601a","Type":"ContainerStarted","Data":"72aa391da2125e406f37f2b6fcbc6e726fd269372cd6fd3cb096202af4cb78cc"} Jul 06 15:57:33.035719 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:33.035583 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:33.035719 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:33.035601 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" event={"ID":"ee2df0eb-9830-477b-9623-2c2763db601a","Type":"ContainerStarted","Data":"5be7d49f91cc208f8f0b6cbc8db5da3984c37f8f74869cf15091db53dfcbeef6"} Jul 06 15:57:33.057216 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:33.057170 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" podStartSLOduration=1.441711055 podStartE2EDuration="2.057155961s" totalImagesPullingTime="615.444906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:57:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:57:33.05513898 +0000 UTC m=+1874.687690963" watchObservedRunningTime="2026-07-06 15:57:33.057155961 +0000 UTC m=+1874.689707944" Jul 06 15:57:35.689993 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:35.689956 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:36.318736 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:36.318699 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:36.420034 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:36.419994 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:36.750659 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:36.750606 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:39.048807 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:39.048774 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:43.218613 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:43.218581 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:43.285098 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:43.285073 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:43.793926 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:43.793897 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:44.094164 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.094083 2561 generic.go:356] "Generic (PLEG): container finished" podID="ee3f6509-7558-4303-8eda-9fbd4f8400c3" containerID="9c590384ec2094cd415ecb69a731e759d10dce9bfc63d5e28a75c3ab6b487baa" exitCode=0 Jul 06 15:57:44.094164 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.094144 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" event={"ID":"ee3f6509-7558-4303-8eda-9fbd4f8400c3","Type":"ContainerDied","Data":"9c590384ec2094cd415ecb69a731e759d10dce9bfc63d5e28a75c3ab6b487baa"} Jul 06 15:57:44.094605 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.094587 2561 scope.go:117] "RemoveContainer" containerID="9c590384ec2094cd415ecb69a731e759d10dce9bfc63d5e28a75c3ab6b487baa" Jul 06 15:57:44.095813 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.095777 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:57:44.095902 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.095866 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:57:44.095945 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.095912 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:57:44.158658 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.158629 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:44.205070 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.205035 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd"] Jul 06 15:57:44.205548 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.205488 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" podUID="ee2df0eb-9830-477b-9623-2c2763db601a" containerName="maas-api" containerID="cri-o://72aa391da2125e406f37f2b6fcbc6e726fd269372cd6fd3cb096202af4cb78cc" gracePeriod=30 Jul 06 15:57:44.230370 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.230334 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:44.465637 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.465610 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:44.578555 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.578491 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khk2m\" (UniqueName: \"kubernetes.io/projected/ee2df0eb-9830-477b-9623-2c2763db601a-kube-api-access-khk2m\") pod \"ee2df0eb-9830-477b-9623-2c2763db601a\" (UID: \"ee2df0eb-9830-477b-9623-2c2763db601a\") " Jul 06 15:57:44.578702 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.578610 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2df0eb-9830-477b-9623-2c2763db601a-openshift-service-ca\") pod \"ee2df0eb-9830-477b-9623-2c2763db601a\" (UID: \"ee2df0eb-9830-477b-9623-2c2763db601a\") " Jul 06 15:57:44.578784 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.578701 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee2df0eb-9830-477b-9623-2c2763db601a-maas-api-tls\") pod \"ee2df0eb-9830-477b-9623-2c2763db601a\" (UID: \"ee2df0eb-9830-477b-9623-2c2763db601a\") " Jul 06 15:57:44.579059 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.579024 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee2df0eb-9830-477b-9623-2c2763db601a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee2df0eb-9830-477b-9623-2c2763db601a" (UID: "ee2df0eb-9830-477b-9623-2c2763db601a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:57:44.580973 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.580932 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2df0eb-9830-477b-9623-2c2763db601a-kube-api-access-khk2m" (OuterVolumeSpecName: "kube-api-access-khk2m") pod "ee2df0eb-9830-477b-9623-2c2763db601a" (UID: "ee2df0eb-9830-477b-9623-2c2763db601a"). InnerVolumeSpecName "kube-api-access-khk2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:57:44.581147 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.581122 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee2df0eb-9830-477b-9623-2c2763db601a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee2df0eb-9830-477b-9623-2c2763db601a" (UID: "ee2df0eb-9830-477b-9623-2c2763db601a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:57:44.679744 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.679677 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khk2m\" (UniqueName: \"kubernetes.io/projected/ee2df0eb-9830-477b-9623-2c2763db601a-kube-api-access-khk2m\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:57:44.679744 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.679702 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee2df0eb-9830-477b-9623-2c2763db601a-openshift-service-ca\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:57:44.679744 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:44.679712 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee2df0eb-9830-477b-9623-2c2763db601a-maas-api-tls\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:57:45.019553 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:45.017070 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:45.100944 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:45.100908 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" event={"ID":"ee3f6509-7558-4303-8eda-9fbd4f8400c3","Type":"ContainerStarted","Data":"29ff0a81cfe00f379fb2713f6891d7667a5cbf6f7ae1d5bb413390e4d27f647c"} Jul 06 15:57:45.102152 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:45.102126 2561 generic.go:356] "Generic (PLEG): container finished" podID="ee2df0eb-9830-477b-9623-2c2763db601a" containerID="72aa391da2125e406f37f2b6fcbc6e726fd269372cd6fd3cb096202af4cb78cc" exitCode=0 Jul 06 15:57:45.102298 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:45.102167 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" Jul 06 15:57:45.102298 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:45.102176 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" event={"ID":"ee2df0eb-9830-477b-9623-2c2763db601a","Type":"ContainerDied","Data":"72aa391da2125e406f37f2b6fcbc6e726fd269372cd6fd3cb096202af4cb78cc"} Jul 06 15:57:45.102298 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:45.102224 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd" event={"ID":"ee2df0eb-9830-477b-9623-2c2763db601a","Type":"ContainerDied","Data":"5be7d49f91cc208f8f0b6cbc8db5da3984c37f8f74869cf15091db53dfcbeef6"} Jul 06 15:57:45.102298 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:45.102251 2561 scope.go:117] "RemoveContainer" containerID="72aa391da2125e406f37f2b6fcbc6e726fd269372cd6fd3cb096202af4cb78cc" Jul 06 15:57:45.120759 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:45.119599 2561 scope.go:117] "RemoveContainer" containerID="72aa391da2125e406f37f2b6fcbc6e726fd269372cd6fd3cb096202af4cb78cc" Jul 06 15:57:45.120759 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:57:45.119994 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72aa391da2125e406f37f2b6fcbc6e726fd269372cd6fd3cb096202af4cb78cc\": container with ID starting with 72aa391da2125e406f37f2b6fcbc6e726fd269372cd6fd3cb096202af4cb78cc not found: ID does not exist" containerID="72aa391da2125e406f37f2b6fcbc6e726fd269372cd6fd3cb096202af4cb78cc" Jul 06 15:57:45.120759 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:45.120031 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72aa391da2125e406f37f2b6fcbc6e726fd269372cd6fd3cb096202af4cb78cc"} err="failed to get container status \"72aa391da2125e406f37f2b6fcbc6e726fd269372cd6fd3cb096202af4cb78cc\": rpc error: code = NotFound desc = could not find container \"72aa391da2125e406f37f2b6fcbc6e726fd269372cd6fd3cb096202af4cb78cc\": container with ID starting with 72aa391da2125e406f37f2b6fcbc6e726fd269372cd6fd3cb096202af4cb78cc not found: ID does not exist" Jul 06 15:57:45.172485 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:45.172453 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd"] Jul 06 15:57:45.178860 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:45.178832 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-37904f1b-78d475cc86-t94sd"] Jul 06 15:57:45.371918 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:45.371887 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:45.373148 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:45.373115 2561 patch_prober.go:28] interesting pod/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 06 15:57:45.373288 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:45.373168 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" podUID="ee3f6509-7558-4303-8eda-9fbd4f8400c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 06 15:57:46.371557 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:46.371498 2561 patch_prober.go:28] interesting pod/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 06 15:57:46.371751 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:46.371608 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" podUID="ee3f6509-7558-4303-8eda-9fbd4f8400c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 06 15:57:46.800492 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:46.800462 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee2df0eb-9830-477b-9623-2c2763db601a" path="/var/lib/kubelet/pods/ee2df0eb-9830-477b-9623-2c2763db601a/volumes" Jul 06 15:57:47.372298 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:47.372264 2561 patch_prober.go:28] interesting pod/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 06 15:57:47.372478 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:47.372327 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" podUID="ee3f6509-7558-4303-8eda-9fbd4f8400c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 06 15:57:48.372090 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:48.372054 2561 patch_prober.go:28] interesting pod/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 06 15:57:48.372515 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:48.372103 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" podUID="ee3f6509-7558-4303-8eda-9fbd4f8400c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 06 15:57:49.375651 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:49.375619 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:49.376017 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:49.375898 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:49.376604 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:49.376585 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:50.890599 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:50.890568 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:50.991703 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:50.991676 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:51.144212 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:51.144130 2561 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" secret="" err="secret \"e2e-mt-37904f1b-openshift-default-dockercfg-t7pf4\" not found" Jul 06 15:57:51.152693 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:51.152667 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd"] Jul 06 15:57:51.174608 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:51.174581 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:51.414802 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:51.414716 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:51.532791 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:51.532759 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:51.628224 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:51.628196 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:51.733391 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:51.733364 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:52.149092 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:52.148971 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" podUID="ee3f6509-7558-4303-8eda-9fbd4f8400c3" containerName="istio-proxy" containerID="cri-o://29ff0a81cfe00f379fb2713f6891d7667a5cbf6f7ae1d5bb413390e4d27f647c" gracePeriod=30 Jul 06 15:57:57.423019 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.422959 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:57.604411 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.604378 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4nnw\" (UniqueName: \"kubernetes.io/projected/ee3f6509-7558-4303-8eda-9fbd4f8400c3-kube-api-access-c4nnw\") pod \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " Jul 06 15:57:57.604661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.604418 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-token\") pod \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " Jul 06 15:57:57.604661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.604455 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-workload-socket\") pod \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " Jul 06 15:57:57.604661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.604476 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istiod-ca-cert\") pod \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " Jul 06 15:57:57.604661 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.604659 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-envoy\") pod \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " Jul 06 15:57:57.604903 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.604685 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-credential-socket\") pod \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " Jul 06 15:57:57.604903 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.604708 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-workload-certs\") pod \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " Jul 06 15:57:57.604903 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.604730 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-data\") pod \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " Jul 06 15:57:57.604903 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.604758 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-podinfo\") pod \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\" (UID: \"ee3f6509-7558-4303-8eda-9fbd4f8400c3\") " Jul 06 15:57:57.604903 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.604832 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee3f6509-7558-4303-8eda-9fbd4f8400c3" (UID: "ee3f6509-7558-4303-8eda-9fbd4f8400c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:57:57.605146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.604936 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee3f6509-7558-4303-8eda-9fbd4f8400c3" (UID: "ee3f6509-7558-4303-8eda-9fbd4f8400c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:57:57.605146 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.604972 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee3f6509-7558-4303-8eda-9fbd4f8400c3" (UID: "ee3f6509-7558-4303-8eda-9fbd4f8400c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:57:57.605421 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.605221 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-workload-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:57:57.605421 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.605243 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istiod-ca-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:57:57.605421 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.605256 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-credential-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:57:57.605421 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.605398 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee3f6509-7558-4303-8eda-9fbd4f8400c3" (UID: "ee3f6509-7558-4303-8eda-9fbd4f8400c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:57:57.605970 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.605944 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee3f6509-7558-4303-8eda-9fbd4f8400c3" (UID: "ee3f6509-7558-4303-8eda-9fbd4f8400c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:57:57.606886 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.606862 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee3f6509-7558-4303-8eda-9fbd4f8400c3" (UID: "ee3f6509-7558-4303-8eda-9fbd4f8400c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:57:57.606985 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.606949 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee3f6509-7558-4303-8eda-9fbd4f8400c3" (UID: "ee3f6509-7558-4303-8eda-9fbd4f8400c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 15:57:57.606985 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.606969 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3f6509-7558-4303-8eda-9fbd4f8400c3-kube-api-access-c4nnw" (OuterVolumeSpecName: "kube-api-access-c4nnw") pod "ee3f6509-7558-4303-8eda-9fbd4f8400c3" (UID: "ee3f6509-7558-4303-8eda-9fbd4f8400c3"). InnerVolumeSpecName "kube-api-access-c4nnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:57:57.607161 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.607136 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee3f6509-7558-4303-8eda-9fbd4f8400c3" (UID: "ee3f6509-7558-4303-8eda-9fbd4f8400c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:57:57.706698 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.706624 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-envoy\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:57:57.706698 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.706659 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-workload-certs\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:57:57.706698 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.706671 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-data\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:57:57.706698 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.706683 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-podinfo\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:57:57.706698 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.706696 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4nnw\" (UniqueName: \"kubernetes.io/projected/ee3f6509-7558-4303-8eda-9fbd4f8400c3-kube-api-access-c4nnw\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:57:57.706927 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:57.706711 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3f6509-7558-4303-8eda-9fbd4f8400c3-istio-token\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:57:58.181614 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.181577 2561 generic.go:356] "Generic (PLEG): container finished" podID="ee3f6509-7558-4303-8eda-9fbd4f8400c3" containerID="29ff0a81cfe00f379fb2713f6891d7667a5cbf6f7ae1d5bb413390e4d27f647c" exitCode=0 Jul 06 15:57:58.181614 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.181617 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" event={"ID":"ee3f6509-7558-4303-8eda-9fbd4f8400c3","Type":"ContainerDied","Data":"29ff0a81cfe00f379fb2713f6891d7667a5cbf6f7ae1d5bb413390e4d27f647c"} Jul 06 15:57:58.181817 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.181660 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" event={"ID":"ee3f6509-7558-4303-8eda-9fbd4f8400c3","Type":"ContainerDied","Data":"e68ccb2b48992fc7224f46c7da7449180f6be61c2250a454f35a882c685adcd4"} Jul 06 15:57:58.181817 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.181681 2561 scope.go:117] "RemoveContainer" containerID="29ff0a81cfe00f379fb2713f6891d7667a5cbf6f7ae1d5bb413390e4d27f647c" Jul 06 15:57:58.181817 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.181619 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd" Jul 06 15:57:58.192466 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.192445 2561 scope.go:117] "RemoveContainer" containerID="9c590384ec2094cd415ecb69a731e759d10dce9bfc63d5e28a75c3ab6b487baa" Jul 06 15:57:58.201697 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.201508 2561 scope.go:117] "RemoveContainer" containerID="29ff0a81cfe00f379fb2713f6891d7667a5cbf6f7ae1d5bb413390e4d27f647c" Jul 06 15:57:58.201897 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:57:58.201878 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ff0a81cfe00f379fb2713f6891d7667a5cbf6f7ae1d5bb413390e4d27f647c\": container with ID starting with 29ff0a81cfe00f379fb2713f6891d7667a5cbf6f7ae1d5bb413390e4d27f647c not found: ID does not exist" containerID="29ff0a81cfe00f379fb2713f6891d7667a5cbf6f7ae1d5bb413390e4d27f647c" Jul 06 15:57:58.201940 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.201910 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ff0a81cfe00f379fb2713f6891d7667a5cbf6f7ae1d5bb413390e4d27f647c"} err="failed to get container status \"29ff0a81cfe00f379fb2713f6891d7667a5cbf6f7ae1d5bb413390e4d27f647c\": rpc error: code = NotFound desc = could not find container \"29ff0a81cfe00f379fb2713f6891d7667a5cbf6f7ae1d5bb413390e4d27f647c\": container with ID starting with 29ff0a81cfe00f379fb2713f6891d7667a5cbf6f7ae1d5bb413390e4d27f647c not found: ID does not exist" Jul 06 15:57:58.201940 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.201928 2561 scope.go:117] "RemoveContainer" containerID="9c590384ec2094cd415ecb69a731e759d10dce9bfc63d5e28a75c3ab6b487baa" Jul 06 15:57:58.202162 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:57:58.202148 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c590384ec2094cd415ecb69a731e759d10dce9bfc63d5e28a75c3ab6b487baa\": container with ID starting with 9c590384ec2094cd415ecb69a731e759d10dce9bfc63d5e28a75c3ab6b487baa not found: ID does not exist" containerID="9c590384ec2094cd415ecb69a731e759d10dce9bfc63d5e28a75c3ab6b487baa" Jul 06 15:57:58.202201 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.202166 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c590384ec2094cd415ecb69a731e759d10dce9bfc63d5e28a75c3ab6b487baa"} err="failed to get container status \"9c590384ec2094cd415ecb69a731e759d10dce9bfc63d5e28a75c3ab6b487baa\": rpc error: code = NotFound desc = could not find container \"9c590384ec2094cd415ecb69a731e759d10dce9bfc63d5e28a75c3ab6b487baa\": container with ID starting with 9c590384ec2094cd415ecb69a731e759d10dce9bfc63d5e28a75c3ab6b487baa not found: ID does not exist" Jul 06 15:57:58.206927 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.206905 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd"] Jul 06 15:57:58.216097 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.216078 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-37904f1b-openshift-default-6fb76d97f5-hhrqd"] Jul 06 15:57:58.362176 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.362149 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:58.438080 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.438020 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:58.534189 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.534155 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:58.632034 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.632006 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:58.733028 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.733008 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:57:58.800445 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:57:58.800418 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee3f6509-7558-4303-8eda-9fbd4f8400c3" path="/var/lib/kubelet/pods/ee3f6509-7558-4303-8eda-9fbd4f8400c3/volumes" Jul 06 15:58:05.148746 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:05.148712 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:05.250362 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:05.250329 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:11.948825 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:11.948788 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:12.178747 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.178356 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv"] Jul 06 15:58:12.178999 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.178977 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee3f6509-7558-4303-8eda-9fbd4f8400c3" containerName="istio-proxy" Jul 06 15:58:12.179079 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.179004 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee3f6509-7558-4303-8eda-9fbd4f8400c3" containerName="istio-proxy" Jul 06 15:58:12.179079 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.179017 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee3f6509-7558-4303-8eda-9fbd4f8400c3" containerName="istio-proxy" Jul 06 15:58:12.179079 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.179025 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee3f6509-7558-4303-8eda-9fbd4f8400c3" containerName="istio-proxy" Jul 06 15:58:12.179079 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.179042 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee2df0eb-9830-477b-9623-2c2763db601a" containerName="maas-api" Jul 06 15:58:12.179079 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.179051 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee2df0eb-9830-477b-9623-2c2763db601a" containerName="maas-api" Jul 06 15:58:12.179277 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.179175 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee3f6509-7558-4303-8eda-9fbd4f8400c3" containerName="istio-proxy" Jul 06 15:58:12.179277 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.179193 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee2df0eb-9830-477b-9623-2c2763db601a" containerName="maas-api" Jul 06 15:58:12.183940 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.183920 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.187028 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.187006 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-37068508-openshift-default-dockercfg-wzcjt\"" Jul 06 15:58:12.194206 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.194180 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv"] Jul 06 15:58:12.210965 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.210881 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:12.231101 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.231079 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-data\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.231210 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.231108 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzbv9\" (UniqueName: \"kubernetes.io/projected/e1fddb7f-f69a-41d5-bc66-eadfa141c611-kube-api-access-kzbv9\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.231257 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.231217 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-token\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.231257 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.231247 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istiod-ca-cert\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.231336 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.231288 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-podinfo\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.231380 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.231356 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-envoy\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.231417 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.231397 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-workload-socket\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.231468 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.231424 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-credential-socket\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.231468 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.231439 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-workload-certs\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.332347 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.332314 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-token\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.332347 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.332349 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istiod-ca-cert\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.332626 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.332389 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-podinfo\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.332626 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.332445 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-envoy\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.332626 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.332485 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-workload-socket\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.332626 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.332514 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-credential-socket\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.332626 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.332565 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-workload-certs\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.332626 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.332595 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-data\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.332626 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.332620 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzbv9\" (UniqueName: \"kubernetes.io/projected/e1fddb7f-f69a-41d5-bc66-eadfa141c611-kube-api-access-kzbv9\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.332989 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.332936 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-workload-socket\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.333155 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.333131 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istiod-ca-cert\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.333304 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.333258 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-workload-certs\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.333304 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.333286 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-data\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.333421 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.333399 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-credential-socket\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.335001 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.334978 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-envoy\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.335213 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.335193 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-podinfo\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.341622 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.341602 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-token\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.341908 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.341889 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzbv9\" (UniqueName: \"kubernetes.io/projected/e1fddb7f-f69a-41d5-bc66-eadfa141c611-kube-api-access-kzbv9\") pod \"e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.495413 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.495381 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:12.648770 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.648746 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv"] Jul 06 15:58:12.651456 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:58:12.651428 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1fddb7f_f69a_41d5_bc66_eadfa141c611.slice/crio-40be1cbcb45196c6998e5f4f41c798be9d5ad496a1a43d27a14a85419566617d WatchSource:0}: Error finding container 40be1cbcb45196c6998e5f4f41c798be9d5ad496a1a43d27a14a85419566617d: Status 404 returned error can't find the container with id 40be1cbcb45196c6998e5f4f41c798be9d5ad496a1a43d27a14a85419566617d Jul 06 15:58:12.654378 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.654334 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:58:12.654456 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.654428 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:58:12.654492 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.654472 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:58:12.751571 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:12.751423 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:13.260928 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:13.260895 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" event={"ID":"e1fddb7f-f69a-41d5-bc66-eadfa141c611","Type":"ContainerStarted","Data":"491754dcc1d77b9cbf9ccc3c8787c3d4b61ee46849d77bd7a95704b308a1d13e"} Jul 06 15:58:13.261382 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:13.260946 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" event={"ID":"e1fddb7f-f69a-41d5-bc66-eadfa141c611","Type":"ContainerStarted","Data":"40be1cbcb45196c6998e5f4f41c798be9d5ad496a1a43d27a14a85419566617d"} Jul 06 15:58:13.287116 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:13.287073 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" podStartSLOduration=1.287058724 podStartE2EDuration="1.287058724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:58:13.283967885 +0000 UTC m=+1914.916519868" watchObservedRunningTime="2026-07-06 15:58:13.287058724 +0000 UTC m=+1914.919610723" Jul 06 15:58:13.495651 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:13.495619 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:13.500633 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:13.500608 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:14.268158 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:14.268130 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:14.269118 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:14.269097 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:15.315222 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.315189 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:15.410330 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.410255 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:15.420980 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.420947 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w"] Jul 06 15:58:15.422000 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.421967 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee3f6509-7558-4303-8eda-9fbd4f8400c3" containerName="istio-proxy" Jul 06 15:58:15.425835 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.425813 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" Jul 06 15:58:15.428456 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.428421 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-37068508-serving-cert\"" Jul 06 15:58:15.428600 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.428519 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rszzk\"" Jul 06 15:58:15.440161 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.440129 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w"] Jul 06 15:58:15.461113 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.461070 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c926390d-37e9-4948-bd31-99bc7d03aeb8-maas-api-tls\") pod \"maas-api-e2e-mt-37068508-66cc997d5-jwz6w\" (UID: \"c926390d-37e9-4948-bd31-99bc7d03aeb8\") " pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" Jul 06 15:58:15.461301 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.461214 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c926390d-37e9-4948-bd31-99bc7d03aeb8-openshift-service-ca\") pod \"maas-api-e2e-mt-37068508-66cc997d5-jwz6w\" (UID: \"c926390d-37e9-4948-bd31-99bc7d03aeb8\") " pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" Jul 06 15:58:15.461301 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.461254 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjdmz\" (UniqueName: \"kubernetes.io/projected/c926390d-37e9-4948-bd31-99bc7d03aeb8-kube-api-access-rjdmz\") pod \"maas-api-e2e-mt-37068508-66cc997d5-jwz6w\" (UID: \"c926390d-37e9-4948-bd31-99bc7d03aeb8\") " pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" Jul 06 15:58:15.511666 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.511635 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:15.562085 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.562053 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c926390d-37e9-4948-bd31-99bc7d03aeb8-maas-api-tls\") pod \"maas-api-e2e-mt-37068508-66cc997d5-jwz6w\" (UID: \"c926390d-37e9-4948-bd31-99bc7d03aeb8\") " pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" Jul 06 15:58:15.562218 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.562140 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c926390d-37e9-4948-bd31-99bc7d03aeb8-openshift-service-ca\") pod \"maas-api-e2e-mt-37068508-66cc997d5-jwz6w\" (UID: \"c926390d-37e9-4948-bd31-99bc7d03aeb8\") " pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" Jul 06 15:58:15.562218 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.562179 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjdmz\" (UniqueName: \"kubernetes.io/projected/c926390d-37e9-4948-bd31-99bc7d03aeb8-kube-api-access-rjdmz\") pod \"maas-api-e2e-mt-37068508-66cc997d5-jwz6w\" (UID: \"c926390d-37e9-4948-bd31-99bc7d03aeb8\") " pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" Jul 06 15:58:15.563190 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.563172 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c926390d-37e9-4948-bd31-99bc7d03aeb8-openshift-service-ca\") pod \"maas-api-e2e-mt-37068508-66cc997d5-jwz6w\" (UID: \"c926390d-37e9-4948-bd31-99bc7d03aeb8\") " pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" Jul 06 15:58:15.565522 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.565446 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c926390d-37e9-4948-bd31-99bc7d03aeb8-maas-api-tls\") pod \"maas-api-e2e-mt-37068508-66cc997d5-jwz6w\" (UID: \"c926390d-37e9-4948-bd31-99bc7d03aeb8\") " pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" Jul 06 15:58:15.572030 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.572007 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjdmz\" (UniqueName: \"kubernetes.io/projected/c926390d-37e9-4948-bd31-99bc7d03aeb8-kube-api-access-rjdmz\") pod \"maas-api-e2e-mt-37068508-66cc997d5-jwz6w\" (UID: \"c926390d-37e9-4948-bd31-99bc7d03aeb8\") " pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" Jul 06 15:58:15.612224 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.612181 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:15.741705 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.741673 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" Jul 06 15:58:15.884096 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:15.884071 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w"] Jul 06 15:58:15.885712 ip-10-0-129-225 kubenswrapper[2561]: W0706 15:58:15.885679 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc926390d_37e9_4948_bd31_99bc7d03aeb8.slice/crio-c0aaca22093b6877dc60cc2f3a45c148e39114c55aa691a2b90ba99aaa985fb4 WatchSource:0}: Error finding container c0aaca22093b6877dc60cc2f3a45c148e39114c55aa691a2b90ba99aaa985fb4: Status 404 returned error can't find the container with id c0aaca22093b6877dc60cc2f3a45c148e39114c55aa691a2b90ba99aaa985fb4 Jul 06 15:58:16.002515 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:16.002470 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:16.103449 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:16.103343 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:16.280703 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:16.280667 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" event={"ID":"c926390d-37e9-4948-bd31-99bc7d03aeb8","Type":"ContainerStarted","Data":"c0aaca22093b6877dc60cc2f3a45c148e39114c55aa691a2b90ba99aaa985fb4"} Jul 06 15:58:17.287236 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:17.287195 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" event={"ID":"c926390d-37e9-4948-bd31-99bc7d03aeb8","Type":"ContainerStarted","Data":"c2ad256c126480c76dbcea75869606fb071f53ef7dc5ee6f4e9a910bd783b050"} Jul 06 15:58:17.287714 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:17.287312 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" Jul 06 15:58:17.309633 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:17.309581 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" podStartSLOduration=1.8600654429999999 podStartE2EDuration="2.309562797s" totalImagesPullingTime="449.497354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:58:17.306864166 +0000 UTC m=+1918.939416150" watchObservedRunningTime="2026-07-06 15:58:17.309562797 +0000 UTC m=+1918.942114779" Jul 06 15:58:19.806706 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:19.806677 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:23.304632 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:23.304600 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" Jul 06 15:58:29.269361 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:29.269323 2561 patch_prober.go:28] interesting pod/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 06 15:58:29.269821 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:29.269388 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" podUID="e1fddb7f-f69a-41d5-bc66-eadfa141c611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 06 15:58:29.366658 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:29.366627 2561 generic.go:356] "Generic (PLEG): container finished" podID="e1fddb7f-f69a-41d5-bc66-eadfa141c611" containerID="491754dcc1d77b9cbf9ccc3c8787c3d4b61ee46849d77bd7a95704b308a1d13e" exitCode=0 Jul 06 15:58:29.366819 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:29.366701 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" event={"ID":"e1fddb7f-f69a-41d5-bc66-eadfa141c611","Type":"ContainerDied","Data":"491754dcc1d77b9cbf9ccc3c8787c3d4b61ee46849d77bd7a95704b308a1d13e"} Jul 06 15:58:29.367109 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:29.367096 2561 scope.go:117] "RemoveContainer" containerID="491754dcc1d77b9cbf9ccc3c8787c3d4b61ee46849d77bd7a95704b308a1d13e" Jul 06 15:58:29.368110 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:29.368085 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:58:29.368184 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:29.368167 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:58:29.368227 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:29.368203 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 15:58:30.373724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:30.373688 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" event={"ID":"e1fddb7f-f69a-41d5-bc66-eadfa141c611","Type":"ContainerStarted","Data":"23db71e5521613b09b1ceb819e1e43b18c47f16dfb6c0d4ea38168a9da14fdc7"} Jul 06 15:58:30.496124 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:30.496089 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:30.496310 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:30.496278 2561 patch_prober.go:28] interesting pod/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 06 15:58:30.496363 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:30.496331 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" podUID="e1fddb7f-f69a-41d5-bc66-eadfa141c611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 06 15:58:31.495741 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:31.495709 2561 patch_prober.go:28] interesting pod/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 06 15:58:31.496114 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:31.495764 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" podUID="e1fddb7f-f69a-41d5-bc66-eadfa141c611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 06 15:58:32.496681 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:32.496647 2561 patch_prober.go:28] interesting pod/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 06 15:58:32.497057 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:32.496699 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" podUID="e1fddb7f-f69a-41d5-bc66-eadfa141c611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 06 15:58:33.496136 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:33.496101 2561 patch_prober.go:28] interesting pod/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 06 15:58:33.496317 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:33.496157 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" podUID="e1fddb7f-f69a-41d5-bc66-eadfa141c611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 06 15:58:34.499953 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:34.499922 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:34.500323 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:34.500270 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:34.500962 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:34.500943 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 15:58:36.677387 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:36.677336 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:57.425511 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:57.425467 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:57.783646 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:57.783616 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w"] Jul 06 15:58:57.783905 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:57.783855 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" podUID="c926390d-37e9-4948-bd31-99bc7d03aeb8" containerName="maas-api" containerID="cri-o://c2ad256c126480c76dbcea75869606fb071f53ef7dc5ee6f4e9a910bd783b050" gracePeriod=30 Jul 06 15:58:57.819441 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:57.819410 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:58.049420 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.049364 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" Jul 06 15:58:58.185385 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.185346 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c926390d-37e9-4948-bd31-99bc7d03aeb8-openshift-service-ca\") pod \"c926390d-37e9-4948-bd31-99bc7d03aeb8\" (UID: \"c926390d-37e9-4948-bd31-99bc7d03aeb8\") " Jul 06 15:58:58.185385 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.185385 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjdmz\" (UniqueName: \"kubernetes.io/projected/c926390d-37e9-4948-bd31-99bc7d03aeb8-kube-api-access-rjdmz\") pod \"c926390d-37e9-4948-bd31-99bc7d03aeb8\" (UID: \"c926390d-37e9-4948-bd31-99bc7d03aeb8\") " Jul 06 15:58:58.185668 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.185412 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c926390d-37e9-4948-bd31-99bc7d03aeb8-maas-api-tls\") pod \"c926390d-37e9-4948-bd31-99bc7d03aeb8\" (UID: \"c926390d-37e9-4948-bd31-99bc7d03aeb8\") " Jul 06 15:58:58.185779 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.185758 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c926390d-37e9-4948-bd31-99bc7d03aeb8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c926390d-37e9-4948-bd31-99bc7d03aeb8" (UID: "c926390d-37e9-4948-bd31-99bc7d03aeb8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:58:58.187767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.187725 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c926390d-37e9-4948-bd31-99bc7d03aeb8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c926390d-37e9-4948-bd31-99bc7d03aeb8" (UID: "c926390d-37e9-4948-bd31-99bc7d03aeb8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:58:58.187767 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.187751 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c926390d-37e9-4948-bd31-99bc7d03aeb8-kube-api-access-rjdmz" (OuterVolumeSpecName: "kube-api-access-rjdmz") pod "c926390d-37e9-4948-bd31-99bc7d03aeb8" (UID: "c926390d-37e9-4948-bd31-99bc7d03aeb8"). InnerVolumeSpecName "kube-api-access-rjdmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:58:58.286983 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.286954 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c926390d-37e9-4948-bd31-99bc7d03aeb8-openshift-service-ca\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:58:58.286983 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.286979 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjdmz\" (UniqueName: \"kubernetes.io/projected/c926390d-37e9-4948-bd31-99bc7d03aeb8-kube-api-access-rjdmz\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:58:58.287165 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.286990 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c926390d-37e9-4948-bd31-99bc7d03aeb8-maas-api-tls\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 15:58:58.519724 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.519696 2561 generic.go:356] "Generic (PLEG): container finished" podID="c926390d-37e9-4948-bd31-99bc7d03aeb8" containerID="c2ad256c126480c76dbcea75869606fb071f53ef7dc5ee6f4e9a910bd783b050" exitCode=0 Jul 06 15:58:58.520127 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.519745 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" Jul 06 15:58:58.520127 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.519788 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" event={"ID":"c926390d-37e9-4948-bd31-99bc7d03aeb8","Type":"ContainerDied","Data":"c2ad256c126480c76dbcea75869606fb071f53ef7dc5ee6f4e9a910bd783b050"} Jul 06 15:58:58.520127 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.519840 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w" event={"ID":"c926390d-37e9-4948-bd31-99bc7d03aeb8","Type":"ContainerDied","Data":"c0aaca22093b6877dc60cc2f3a45c148e39114c55aa691a2b90ba99aaa985fb4"} Jul 06 15:58:58.520127 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.519863 2561 scope.go:117] "RemoveContainer" containerID="c2ad256c126480c76dbcea75869606fb071f53ef7dc5ee6f4e9a910bd783b050" Jul 06 15:58:58.529376 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.529359 2561 scope.go:117] "RemoveContainer" containerID="c2ad256c126480c76dbcea75869606fb071f53ef7dc5ee6f4e9a910bd783b050" Jul 06 15:58:58.529644 ip-10-0-129-225 kubenswrapper[2561]: E0706 15:58:58.529625 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ad256c126480c76dbcea75869606fb071f53ef7dc5ee6f4e9a910bd783b050\": container with ID starting with c2ad256c126480c76dbcea75869606fb071f53ef7dc5ee6f4e9a910bd783b050 not found: ID does not exist" containerID="c2ad256c126480c76dbcea75869606fb071f53ef7dc5ee6f4e9a910bd783b050" Jul 06 15:58:58.529696 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.529653 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ad256c126480c76dbcea75869606fb071f53ef7dc5ee6f4e9a910bd783b050"} err="failed to get container status \"c2ad256c126480c76dbcea75869606fb071f53ef7dc5ee6f4e9a910bd783b050\": rpc error: code = NotFound desc = could not find container \"c2ad256c126480c76dbcea75869606fb071f53ef7dc5ee6f4e9a910bd783b050\": container with ID starting with c2ad256c126480c76dbcea75869606fb071f53ef7dc5ee6f4e9a910bd783b050 not found: ID does not exist" Jul 06 15:58:58.542794 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.542774 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w"] Jul 06 15:58:58.547073 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.547055 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-37068508-66cc997d5-jwz6w"] Jul 06 15:58:58.801852 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:58:58.801776 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c926390d-37e9-4948-bd31-99bc7d03aeb8" path="/var/lib/kubelet/pods/c926390d-37e9-4948-bd31-99bc7d03aeb8/volumes" Jul 06 15:59:03.547185 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:59:03.547150 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:08.641542 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:59:08.641492 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:45.500722 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:59:45.500692 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:46.034658 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:59:46.034622 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:51.899641 ip-10-0-129-225 kubenswrapper[2561]: I0706 15:59:51.899610 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:00.142192 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:00.142160 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722560-skpg7"] Jul 06 16:00:00.142598 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:00.142582 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c926390d-37e9-4948-bd31-99bc7d03aeb8" containerName="maas-api" Jul 06 16:00:00.142652 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:00.142600 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c926390d-37e9-4948-bd31-99bc7d03aeb8" containerName="maas-api" Jul 06 16:00:00.142700 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:00.142691 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c926390d-37e9-4948-bd31-99bc7d03aeb8" containerName="maas-api" Jul 06 16:00:00.145813 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:00.145797 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722560-skpg7" Jul 06 16:00:00.149028 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:00.149003 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rszzk\"" Jul 06 16:00:00.156930 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:00.156019 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722560-skpg7"] Jul 06 16:00:00.252118 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:00.252085 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xchh\" (UniqueName: \"kubernetes.io/projected/8d951172-87e6-472e-a117-9210a1a7f20a-kube-api-access-2xchh\") pod \"maas-api-key-cleanup-29722560-skpg7\" (UID: \"8d951172-87e6-472e-a117-9210a1a7f20a\") " pod="opendatahub/maas-api-key-cleanup-29722560-skpg7" Jul 06 16:00:00.353231 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:00.353205 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xchh\" (UniqueName: \"kubernetes.io/projected/8d951172-87e6-472e-a117-9210a1a7f20a-kube-api-access-2xchh\") pod \"maas-api-key-cleanup-29722560-skpg7\" (UID: \"8d951172-87e6-472e-a117-9210a1a7f20a\") " pod="opendatahub/maas-api-key-cleanup-29722560-skpg7" Jul 06 16:00:00.363093 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:00.363061 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xchh\" (UniqueName: \"kubernetes.io/projected/8d951172-87e6-472e-a117-9210a1a7f20a-kube-api-access-2xchh\") pod \"maas-api-key-cleanup-29722560-skpg7\" (UID: \"8d951172-87e6-472e-a117-9210a1a7f20a\") " pod="opendatahub/maas-api-key-cleanup-29722560-skpg7" Jul 06 16:00:00.465871 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:00.465813 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722560-skpg7" Jul 06 16:00:00.607326 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:00.607299 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722560-skpg7"] Jul 06 16:00:00.608011 ip-10-0-129-225 kubenswrapper[2561]: W0706 16:00:00.607988 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d951172_87e6_472e_a117_9210a1a7f20a.slice/crio-a409e0f64f6e81440d4467cf339f72d746d5066b42de6c7d703496f78e618fd9 WatchSource:0}: Error finding container a409e0f64f6e81440d4467cf339f72d746d5066b42de6c7d703496f78e618fd9: Status 404 returned error can't find the container with id a409e0f64f6e81440d4467cf339f72d746d5066b42de6c7d703496f78e618fd9 Jul 06 16:00:00.848533 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:00.848501 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722560-skpg7" event={"ID":"8d951172-87e6-472e-a117-9210a1a7f20a","Type":"ContainerStarted","Data":"575d6d558a12c23c9d90956b005807d69159f4a065089d1333a978045fb226da"} Jul 06 16:00:00.848657 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:00.848574 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722560-skpg7" event={"ID":"8d951172-87e6-472e-a117-9210a1a7f20a","Type":"ContainerStarted","Data":"a409e0f64f6e81440d4467cf339f72d746d5066b42de6c7d703496f78e618fd9"} Jul 06 16:00:00.868025 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:00.867979 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722560-skpg7" podStartSLOduration=0.867964723 podStartE2EDuration="867.964723ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:00:00.866383276 +0000 UTC m=+2022.498935261" watchObservedRunningTime="2026-07-06 16:00:00.867964723 +0000 UTC m=+2022.500516706" Jul 06 16:00:01.854717 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:01.854685 2561 generic.go:356] "Generic (PLEG): container finished" podID="8d951172-87e6-472e-a117-9210a1a7f20a" containerID="575d6d558a12c23c9d90956b005807d69159f4a065089d1333a978045fb226da" exitCode=0 Jul 06 16:00:01.855087 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:01.854777 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722560-skpg7" event={"ID":"8d951172-87e6-472e-a117-9210a1a7f20a","Type":"ContainerDied","Data":"575d6d558a12c23c9d90956b005807d69159f4a065089d1333a978045fb226da"} Jul 06 16:00:03.002243 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:03.002219 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722560-skpg7" Jul 06 16:00:03.184774 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:03.184695 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xchh\" (UniqueName: \"kubernetes.io/projected/8d951172-87e6-472e-a117-9210a1a7f20a-kube-api-access-2xchh\") pod \"8d951172-87e6-472e-a117-9210a1a7f20a\" (UID: \"8d951172-87e6-472e-a117-9210a1a7f20a\") " Jul 06 16:00:03.186897 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:03.186876 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d951172-87e6-472e-a117-9210a1a7f20a-kube-api-access-2xchh" (OuterVolumeSpecName: "kube-api-access-2xchh") pod "8d951172-87e6-472e-a117-9210a1a7f20a" (UID: "8d951172-87e6-472e-a117-9210a1a7f20a"). InnerVolumeSpecName "kube-api-access-2xchh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:00:03.285721 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:03.285692 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xchh\" (UniqueName: \"kubernetes.io/projected/8d951172-87e6-472e-a117-9210a1a7f20a-kube-api-access-2xchh\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:00:03.866328 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:03.866297 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722560-skpg7" event={"ID":"8d951172-87e6-472e-a117-9210a1a7f20a","Type":"ContainerDied","Data":"a409e0f64f6e81440d4467cf339f72d746d5066b42de6c7d703496f78e618fd9"} Jul 06 16:00:03.866502 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:03.866338 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a409e0f64f6e81440d4467cf339f72d746d5066b42de6c7d703496f78e618fd9" Jul 06 16:00:03.866502 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:03.866308 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722560-skpg7" Jul 06 16:00:07.357177 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:07.357139 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:07.635076 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:07.634993 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv"] Jul 06 16:00:07.635256 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:07.635231 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" podUID="e1fddb7f-f69a-41d5-bc66-eadfa141c611" containerName="istio-proxy" containerID="cri-o://23db71e5521613b09b1ceb819e1e43b18c47f16dfb6c0d4ea38168a9da14fdc7" gracePeriod=30 Jul 06 16:00:10.553869 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:10.553839 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:11.630147 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:11.630116 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:12.876822 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.876794 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 16:00:12.922633 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.922490 2561 generic.go:356] "Generic (PLEG): container finished" podID="e1fddb7f-f69a-41d5-bc66-eadfa141c611" containerID="23db71e5521613b09b1ceb819e1e43b18c47f16dfb6c0d4ea38168a9da14fdc7" exitCode=0 Jul 06 16:00:12.922633 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.922557 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" Jul 06 16:00:12.922633 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.922576 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" event={"ID":"e1fddb7f-f69a-41d5-bc66-eadfa141c611","Type":"ContainerDied","Data":"23db71e5521613b09b1ceb819e1e43b18c47f16dfb6c0d4ea38168a9da14fdc7"} Jul 06 16:00:12.922633 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.922633 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv" event={"ID":"e1fddb7f-f69a-41d5-bc66-eadfa141c611","Type":"ContainerDied","Data":"40be1cbcb45196c6998e5f4f41c798be9d5ad496a1a43d27a14a85419566617d"} Jul 06 16:00:12.922965 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.922665 2561 scope.go:117] "RemoveContainer" containerID="23db71e5521613b09b1ceb819e1e43b18c47f16dfb6c0d4ea38168a9da14fdc7" Jul 06 16:00:12.931881 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.931858 2561 scope.go:117] "RemoveContainer" containerID="491754dcc1d77b9cbf9ccc3c8787c3d4b61ee46849d77bd7a95704b308a1d13e" Jul 06 16:00:12.940065 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.940041 2561 scope.go:117] "RemoveContainer" containerID="23db71e5521613b09b1ceb819e1e43b18c47f16dfb6c0d4ea38168a9da14fdc7" Jul 06 16:00:12.940282 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:00:12.940266 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23db71e5521613b09b1ceb819e1e43b18c47f16dfb6c0d4ea38168a9da14fdc7\": container with ID starting with 23db71e5521613b09b1ceb819e1e43b18c47f16dfb6c0d4ea38168a9da14fdc7 not found: ID does not exist" containerID="23db71e5521613b09b1ceb819e1e43b18c47f16dfb6c0d4ea38168a9da14fdc7" Jul 06 16:00:12.940353 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.940296 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23db71e5521613b09b1ceb819e1e43b18c47f16dfb6c0d4ea38168a9da14fdc7"} err="failed to get container status \"23db71e5521613b09b1ceb819e1e43b18c47f16dfb6c0d4ea38168a9da14fdc7\": rpc error: code = NotFound desc = could not find container \"23db71e5521613b09b1ceb819e1e43b18c47f16dfb6c0d4ea38168a9da14fdc7\": container with ID starting with 23db71e5521613b09b1ceb819e1e43b18c47f16dfb6c0d4ea38168a9da14fdc7 not found: ID does not exist" Jul 06 16:00:12.940353 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.940314 2561 scope.go:117] "RemoveContainer" containerID="491754dcc1d77b9cbf9ccc3c8787c3d4b61ee46849d77bd7a95704b308a1d13e" Jul 06 16:00:12.940598 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:00:12.940580 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491754dcc1d77b9cbf9ccc3c8787c3d4b61ee46849d77bd7a95704b308a1d13e\": container with ID starting with 491754dcc1d77b9cbf9ccc3c8787c3d4b61ee46849d77bd7a95704b308a1d13e not found: ID does not exist" containerID="491754dcc1d77b9cbf9ccc3c8787c3d4b61ee46849d77bd7a95704b308a1d13e" Jul 06 16:00:12.940670 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.940606 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491754dcc1d77b9cbf9ccc3c8787c3d4b61ee46849d77bd7a95704b308a1d13e"} err="failed to get container status \"491754dcc1d77b9cbf9ccc3c8787c3d4b61ee46849d77bd7a95704b308a1d13e\": rpc error: code = NotFound desc = could not find container \"491754dcc1d77b9cbf9ccc3c8787c3d4b61ee46849d77bd7a95704b308a1d13e\": container with ID starting with 491754dcc1d77b9cbf9ccc3c8787c3d4b61ee46849d77bd7a95704b308a1d13e not found: ID does not exist" Jul 06 16:00:12.974864 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.974834 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-credential-socket\") pod \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " Jul 06 16:00:12.974950 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.974877 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-envoy\") pod \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " Jul 06 16:00:12.974950 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.974907 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzbv9\" (UniqueName: \"kubernetes.io/projected/e1fddb7f-f69a-41d5-bc66-eadfa141c611-kube-api-access-kzbv9\") pod \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " Jul 06 16:00:12.974950 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.974942 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istiod-ca-cert\") pod \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " Jul 06 16:00:12.975072 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.974974 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-data\") pod \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " Jul 06 16:00:12.975072 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.975011 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-workload-socket\") pod \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " Jul 06 16:00:12.975072 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.975053 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-token\") pod \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " Jul 06 16:00:12.975219 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.975091 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-podinfo\") pod \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " Jul 06 16:00:12.975219 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.975136 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-workload-certs\") pod \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\" (UID: \"e1fddb7f-f69a-41d5-bc66-eadfa141c611\") " Jul 06 16:00:12.975219 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.975164 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e1fddb7f-f69a-41d5-bc66-eadfa141c611" (UID: "e1fddb7f-f69a-41d5-bc66-eadfa141c611"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:00:12.975375 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.975337 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e1fddb7f-f69a-41d5-bc66-eadfa141c611" (UID: "e1fddb7f-f69a-41d5-bc66-eadfa141c611"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:00:12.975423 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.975367 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e1fddb7f-f69a-41d5-bc66-eadfa141c611" (UID: "e1fddb7f-f69a-41d5-bc66-eadfa141c611"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:00:12.975637 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.975616 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e1fddb7f-f69a-41d5-bc66-eadfa141c611" (UID: "e1fddb7f-f69a-41d5-bc66-eadfa141c611"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:00:12.975893 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.975873 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-workload-certs\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:00:12.975937 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.975902 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-credential-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:00:12.975937 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.975918 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istiod-ca-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:00:12.975937 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.975932 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-workload-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:00:12.976360 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.976325 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-data" (OuterVolumeSpecName: "istio-data") pod "e1fddb7f-f69a-41d5-bc66-eadfa141c611" (UID: "e1fddb7f-f69a-41d5-bc66-eadfa141c611"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:00:12.977295 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.977265 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1fddb7f-f69a-41d5-bc66-eadfa141c611-kube-api-access-kzbv9" (OuterVolumeSpecName: "kube-api-access-kzbv9") pod "e1fddb7f-f69a-41d5-bc66-eadfa141c611" (UID: "e1fddb7f-f69a-41d5-bc66-eadfa141c611"). InnerVolumeSpecName "kube-api-access-kzbv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:00:12.977295 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.977288 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e1fddb7f-f69a-41d5-bc66-eadfa141c611" (UID: "e1fddb7f-f69a-41d5-bc66-eadfa141c611"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:00:12.977455 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.977352 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-token" (OuterVolumeSpecName: "istio-token") pod "e1fddb7f-f69a-41d5-bc66-eadfa141c611" (UID: "e1fddb7f-f69a-41d5-bc66-eadfa141c611"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:00:12.977669 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:12.977646 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e1fddb7f-f69a-41d5-bc66-eadfa141c611" (UID: "e1fddb7f-f69a-41d5-bc66-eadfa141c611"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:00:13.076971 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:13.076942 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-data\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:00:13.076971 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:13.076966 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-token\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:00:13.076971 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:13.076975 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-podinfo\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:00:13.077152 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:13.076983 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1fddb7f-f69a-41d5-bc66-eadfa141c611-istio-envoy\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:00:13.077152 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:13.076991 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzbv9\" (UniqueName: \"kubernetes.io/projected/e1fddb7f-f69a-41d5-bc66-eadfa141c611-kube-api-access-kzbv9\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:00:13.246141 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:13.246112 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv"] Jul 06 16:00:13.251148 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:13.251121 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-37068508-openshift-default-64dfd688cf-cl5pv"] Jul 06 16:00:14.799626 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:14.799593 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1fddb7f-f69a-41d5-bc66-eadfa141c611" path="/var/lib/kubelet/pods/e1fddb7f-f69a-41d5-bc66-eadfa141c611/volumes" Jul 06 16:00:18.669546 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:18.669493 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:18.772017 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:18.771989 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:19.166758 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:19.166726 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:26.370242 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:26.370209 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:26.472077 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:26.472043 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:36.731403 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:36.731367 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:37.076345 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:37.076258 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:43.393904 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:43.393830 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:43.494044 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:43.494005 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:00:58.094378 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:00:58.094345 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:04.802494 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:04.802459 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:04.871328 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:04.871282 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:05.153710 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.153630 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:05.426805 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.426697 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb"] Jul 06 16:01:05.427347 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.427322 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1fddb7f-f69a-41d5-bc66-eadfa141c611" containerName="istio-proxy" Jul 06 16:01:05.427347 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.427344 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1fddb7f-f69a-41d5-bc66-eadfa141c611" containerName="istio-proxy" Jul 06 16:01:05.427572 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.427361 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d951172-87e6-472e-a117-9210a1a7f20a" containerName="cleanup" Jul 06 16:01:05.427572 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.427367 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d951172-87e6-472e-a117-9210a1a7f20a" containerName="cleanup" Jul 06 16:01:05.427572 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.427462 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1fddb7f-f69a-41d5-bc66-eadfa141c611" containerName="istio-proxy" Jul 06 16:01:05.427572 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.427487 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1fddb7f-f69a-41d5-bc66-eadfa141c611" containerName="istio-proxy" Jul 06 16:01:05.427572 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.427494 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d951172-87e6-472e-a117-9210a1a7f20a" containerName="cleanup" Jul 06 16:01:05.431296 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.431263 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.434659 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.434637 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-cff200-openshift-default-dockercfg-btc64\"" Jul 06 16:01:05.443822 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.443800 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb"] Jul 06 16:01:05.458081 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.458060 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:05.528574 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.528520 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-data\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.528740 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.528605 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-workload-socket\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.528740 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.528635 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-workload-certs\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.528740 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.528703 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-credential-socket\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.528740 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.528733 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-envoy\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.528974 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.528752 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-podinfo\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.528974 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.528774 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de13cf94-1748-46b8-bd3e-55a45b65c56e-istiod-ca-cert\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.528974 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.528798 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-token\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.528974 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.528835 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6pcd\" (UniqueName: \"kubernetes.io/projected/de13cf94-1748-46b8-bd3e-55a45b65c56e-kube-api-access-p6pcd\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.629544 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.629481 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-workload-socket\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.629723 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.629557 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-workload-certs\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.629798 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.629666 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-credential-socket\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.629864 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.629802 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-envoy\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.629864 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.629842 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-podinfo\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.629977 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.629882 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de13cf94-1748-46b8-bd3e-55a45b65c56e-istiod-ca-cert\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.629977 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.629914 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-token\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.629977 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.629946 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-workload-socket\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.629977 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.629969 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6pcd\" (UniqueName: \"kubernetes.io/projected/de13cf94-1748-46b8-bd3e-55a45b65c56e-kube-api-access-p6pcd\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.630179 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.629984 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-workload-certs\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.630179 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.630020 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-data\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.630390 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.630362 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-data\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.630658 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.630638 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de13cf94-1748-46b8-bd3e-55a45b65c56e-istiod-ca-cert\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.630760 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.630738 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-credential-socket\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.632774 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.632753 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-envoy\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.633071 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.633048 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-podinfo\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.638933 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.638911 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-token\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.639133 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.639115 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6pcd\" (UniqueName: \"kubernetes.io/projected/de13cf94-1748-46b8-bd3e-55a45b65c56e-kube-api-access-p6pcd\") pod \"e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.744477 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.744454 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:05.840363 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.840334 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:05.895585 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.895560 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb"] Jul 06 16:01:05.897707 ip-10-0-129-225 kubenswrapper[2561]: W0706 16:01:05.897646 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde13cf94_1748_46b8_bd3e_55a45b65c56e.slice/crio-a8ed80f23e31d4767ee2698d47c86a865e73fa7911be9cd3298b29ce9db5edf3 WatchSource:0}: Error finding container a8ed80f23e31d4767ee2698d47c86a865e73fa7911be9cd3298b29ce9db5edf3: Status 404 returned error can't find the container with id a8ed80f23e31d4767ee2698d47c86a865e73fa7911be9cd3298b29ce9db5edf3 Jul 06 16:01:05.900564 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.900505 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:01:05.900679 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.900597 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:01:05.900679 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:05.900627 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:01:06.191027 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:06.190934 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" event={"ID":"de13cf94-1748-46b8-bd3e-55a45b65c56e","Type":"ContainerStarted","Data":"fb1f8b2008b5fe78273580cfcc0407be94a54b75b2c7eded77291644b18e3aac"} Jul 06 16:01:06.191027 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:06.190992 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" event={"ID":"de13cf94-1748-46b8-bd3e-55a45b65c56e","Type":"ContainerStarted","Data":"a8ed80f23e31d4767ee2698d47c86a865e73fa7911be9cd3298b29ce9db5edf3"} Jul 06 16:01:06.215487 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:06.215433 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" podStartSLOduration=1.21540971 podStartE2EDuration="1.21540971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:01:06.21487188 +0000 UTC m=+2087.847423864" watchObservedRunningTime="2026-07-06 16:01:06.21540971 +0000 UTC m=+2087.847961695" Jul 06 16:01:06.745235 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:06.745205 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:06.750170 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:06.750146 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:07.196140 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:07.196059 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:07.197022 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:07.197005 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:08.280382 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.280338 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:08.377903 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.377875 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:08.493757 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.493726 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm"] Jul 06 16:01:08.494242 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.494224 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1fddb7f-f69a-41d5-bc66-eadfa141c611" containerName="istio-proxy" Jul 06 16:01:08.494242 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.494244 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1fddb7f-f69a-41d5-bc66-eadfa141c611" containerName="istio-proxy" Jul 06 16:01:08.500022 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.499994 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:01:08.502784 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.502762 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-cff200-serving-cert\"" Jul 06 16:01:08.502955 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.502938 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rszzk\"" Jul 06 16:01:08.511321 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.511296 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm"] Jul 06 16:01:08.557641 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.557577 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3be9e52-8861-471a-8969-cce362a5161d-maas-api-tls\") pod \"maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm\" (UID: \"b3be9e52-8861-471a-8969-cce362a5161d\") " pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:01:08.557641 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.557632 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9bbk\" (UniqueName: \"kubernetes.io/projected/b3be9e52-8861-471a-8969-cce362a5161d-kube-api-access-w9bbk\") pod \"maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm\" (UID: \"b3be9e52-8861-471a-8969-cce362a5161d\") " pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:01:08.557836 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.557731 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3be9e52-8861-471a-8969-cce362a5161d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm\" (UID: \"b3be9e52-8861-471a-8969-cce362a5161d\") " pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:01:08.658822 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.658789 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3be9e52-8861-471a-8969-cce362a5161d-maas-api-tls\") pod \"maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm\" (UID: \"b3be9e52-8861-471a-8969-cce362a5161d\") " pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:01:08.658998 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.658866 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9bbk\" (UniqueName: \"kubernetes.io/projected/b3be9e52-8861-471a-8969-cce362a5161d-kube-api-access-w9bbk\") pod \"maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm\" (UID: \"b3be9e52-8861-471a-8969-cce362a5161d\") " pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:01:08.658998 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.658917 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3be9e52-8861-471a-8969-cce362a5161d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm\" (UID: \"b3be9e52-8861-471a-8969-cce362a5161d\") " pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:01:08.658998 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:01:08.658950 2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-cff200-serving-cert: secret "maas-api-e2e-inf-a-cff200-serving-cert" not found Jul 06 16:01:08.659138 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:01:08.659037 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3be9e52-8861-471a-8969-cce362a5161d-maas-api-tls podName:b3be9e52-8861-471a-8969-cce362a5161d nodeName:}" failed. No retries permitted until 2026-07-06 16:01:09.159018266 +0000 UTC m=+2090.791570227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3be9e52-8861-471a-8969-cce362a5161d-maas-api-tls") pod "maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" (UID: "b3be9e52-8861-471a-8969-cce362a5161d") : secret "maas-api-e2e-inf-a-cff200-serving-cert" not found Jul 06 16:01:08.659730 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.659711 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3be9e52-8861-471a-8969-cce362a5161d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm\" (UID: \"b3be9e52-8861-471a-8969-cce362a5161d\") " pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:01:08.669325 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:08.669298 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9bbk\" (UniqueName: \"kubernetes.io/projected/b3be9e52-8861-471a-8969-cce362a5161d-kube-api-access-w9bbk\") pod \"maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm\" (UID: \"b3be9e52-8861-471a-8969-cce362a5161d\") " pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:01:09.041827 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:09.041795 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:09.143521 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:09.143483 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:09.164125 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:09.164098 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3be9e52-8861-471a-8969-cce362a5161d-maas-api-tls\") pod \"maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm\" (UID: \"b3be9e52-8861-471a-8969-cce362a5161d\") " pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:01:09.166587 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:09.166566 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3be9e52-8861-471a-8969-cce362a5161d-maas-api-tls\") pod \"maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm\" (UID: \"b3be9e52-8861-471a-8969-cce362a5161d\") " pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:01:09.415426 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:09.415348 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:01:09.568868 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:09.568842 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm"] Jul 06 16:01:09.570912 ip-10-0-129-225 kubenswrapper[2561]: W0706 16:01:09.570883 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3be9e52_8861_471a_8969_cce362a5161d.slice/crio-bd9d76f4c31cc3bfff84c2c19ecddea6e6e17312cb1f587203a7041de30a017d WatchSource:0}: Error finding container bd9d76f4c31cc3bfff84c2c19ecddea6e6e17312cb1f587203a7041de30a017d: Status 404 returned error can't find the container with id bd9d76f4c31cc3bfff84c2c19ecddea6e6e17312cb1f587203a7041de30a017d Jul 06 16:01:09.572291 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:09.572265 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 16:01:09.824641 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:09.824610 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:10.051586 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:10.051554 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:10.216682 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:10.216646 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" event={"ID":"b3be9e52-8861-471a-8969-cce362a5161d","Type":"ContainerStarted","Data":"bd9d76f4c31cc3bfff84c2c19ecddea6e6e17312cb1f587203a7041de30a017d"} Jul 06 16:01:11.223014 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:11.222978 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" event={"ID":"b3be9e52-8861-471a-8969-cce362a5161d","Type":"ContainerStarted","Data":"8214817e281ed3a988bceab14916c0c191f6c2ce55b0864d2716d8c9937d09f9"} Jul 06 16:01:11.223504 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:11.223056 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:01:11.243797 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:11.243754 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" podStartSLOduration=2.643784022 podStartE2EDuration="3.243741147s" totalImagesPullingTime="599.957125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:01:11.241709237 +0000 UTC m=+2092.874261222" watchObservedRunningTime="2026-07-06 16:01:11.243741147 +0000 UTC m=+2092.876293130" Jul 06 16:01:13.341218 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.341185 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:13.424877 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.424831 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:13.445324 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.445287 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn"] Jul 06 16:01:13.450794 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.450469 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.452392 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.452361 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn"] Jul 06 16:01:13.453852 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.453830 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-57aaf3-openshift-default-dockercfg-xhc5x\"" Jul 06 16:01:13.509552 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.507586 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-podinfo\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.509552 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.507651 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-credential-socket\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.509552 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.507681 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istiod-ca-cert\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.509552 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.507742 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-envoy\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.509552 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.507779 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-workload-socket\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.509552 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.507807 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-token\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.509552 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.507856 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsjq7\" (UniqueName: \"kubernetes.io/projected/38cd3ea0-a846-4a9d-9797-bfe21d492b56-kube-api-access-wsjq7\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.509552 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.507902 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-workload-certs\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.509552 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.507958 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-data\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.520550 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.516231 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:13.609388 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.609296 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-data\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.609588 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.609391 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-podinfo\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.609588 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.609432 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-credential-socket\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.609588 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.609462 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istiod-ca-cert\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.609588 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.609541 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-envoy\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.609588 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.609582 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-workload-socket\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.609870 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.609610 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-token\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.609870 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.609658 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsjq7\" (UniqueName: \"kubernetes.io/projected/38cd3ea0-a846-4a9d-9797-bfe21d492b56-kube-api-access-wsjq7\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.609870 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.609705 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-workload-certs\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.610083 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.610055 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-workload-certs\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.610150 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.610130 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-data\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.610333 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.610306 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istiod-ca-cert\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.613551 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.610678 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-workload-socket\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.613551 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.610857 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-credential-socket\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.613551 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.612794 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-envoy\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.613551 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.613266 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-podinfo\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.621550 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.620400 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsjq7\" (UniqueName: \"kubernetes.io/projected/38cd3ea0-a846-4a9d-9797-bfe21d492b56-kube-api-access-wsjq7\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.621550 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.620626 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-token\") pod \"e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.762734 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.762705 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:13.976998 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.976967 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn"] Jul 06 16:01:13.980036 ip-10-0-129-225 kubenswrapper[2561]: W0706 16:01:13.980011 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38cd3ea0_a846_4a9d_9797_bfe21d492b56.slice/crio-09fae4df9535b5a9cf69fcd1b75d1d0eade31351362c5b553148b38b83eac636 WatchSource:0}: Error finding container 09fae4df9535b5a9cf69fcd1b75d1d0eade31351362c5b553148b38b83eac636: Status 404 returned error can't find the container with id 09fae4df9535b5a9cf69fcd1b75d1d0eade31351362c5b553148b38b83eac636 Jul 06 16:01:13.982365 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.982334 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:01:13.982447 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.982418 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:01:13.982516 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:13.982456 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:01:14.053589 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:14.053517 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:14.242978 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:14.242936 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" event={"ID":"38cd3ea0-a846-4a9d-9797-bfe21d492b56","Type":"ContainerStarted","Data":"a73932a5dd693825cd1eefbfe7abb27633475e0ed405227562d3083526e967bf"} Jul 06 16:01:14.243173 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:14.243005 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" event={"ID":"38cd3ea0-a846-4a9d-9797-bfe21d492b56","Type":"ContainerStarted","Data":"09fae4df9535b5a9cf69fcd1b75d1d0eade31351362c5b553148b38b83eac636"} Jul 06 16:01:14.266056 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:14.266013 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" podStartSLOduration=1.26599711 podStartE2EDuration="1.26599711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:01:14.264997444 +0000 UTC m=+2095.897549437" watchObservedRunningTime="2026-07-06 16:01:14.26599711 +0000 UTC m=+2095.898549093" Jul 06 16:01:14.763241 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:14.763214 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:15.768415 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:15.768376 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:16.254173 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:16.254144 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:16.255202 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:16.255181 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:16.288687 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:16.288663 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:16.390719 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:16.390686 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:16.489814 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:16.489781 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:16.904749 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:16.904715 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:17.069341 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:17.069305 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:17.171996 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:17.171913 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:17.238830 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:17.238801 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:01:18.909938 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:18.909911 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 16:01:18.911719 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:18.911631 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 16:01:20.785969 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:20.785938 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:21.283254 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:21.283225 2561 generic.go:356] "Generic (PLEG): container finished" podID="de13cf94-1748-46b8-bd3e-55a45b65c56e" containerID="fb1f8b2008b5fe78273580cfcc0407be94a54b75b2c7eded77291644b18e3aac" exitCode=0 Jul 06 16:01:21.283460 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:21.283283 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" event={"ID":"de13cf94-1748-46b8-bd3e-55a45b65c56e","Type":"ContainerDied","Data":"fb1f8b2008b5fe78273580cfcc0407be94a54b75b2c7eded77291644b18e3aac"} Jul 06 16:01:21.283748 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:21.283725 2561 scope.go:117] "RemoveContainer" containerID="fb1f8b2008b5fe78273580cfcc0407be94a54b75b2c7eded77291644b18e3aac" Jul 06 16:01:21.284811 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:21.284784 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:01:21.284891 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:21.284850 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:01:21.284891 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:21.284884 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:01:22.196198 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:22.196166 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:22.290125 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:22.290083 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" event={"ID":"de13cf94-1748-46b8-bd3e-55a45b65c56e","Type":"ContainerStarted","Data":"f0f3488de113b82b3c478a26b89722e250e50ea0835248b8d293dea4ff9afcfb"} Jul 06 16:01:22.746326 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:22.746290 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:22.746742 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:22.746721 2561 patch_prober.go:28] interesting pod/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 06 16:01:22.746903 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:22.746880 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" podUID="de13cf94-1748-46b8-bd3e-55a45b65c56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 06 16:01:23.745613 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:23.745574 2561 patch_prober.go:28] interesting pod/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 06 16:01:23.746194 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:23.746162 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" podUID="de13cf94-1748-46b8-bd3e-55a45b65c56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 06 16:01:24.751665 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:24.751623 2561 patch_prober.go:28] interesting pod/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 06 16:01:24.752140 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:24.751685 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" podUID="de13cf94-1748-46b8-bd3e-55a45b65c56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 06 16:01:25.745471 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:25.745437 2561 patch_prober.go:28] interesting pod/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 06 16:01:25.745695 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:25.745498 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" podUID="de13cf94-1748-46b8-bd3e-55a45b65c56e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 06 16:01:26.750576 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:26.750514 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:26.751012 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:26.750829 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:26.751488 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:26.751471 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:01:28.336027 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:28.335956 2561 generic.go:356] "Generic (PLEG): container finished" podID="38cd3ea0-a846-4a9d-9797-bfe21d492b56" containerID="a73932a5dd693825cd1eefbfe7abb27633475e0ed405227562d3083526e967bf" exitCode=0 Jul 06 16:01:28.336418 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:28.336029 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" event={"ID":"38cd3ea0-a846-4a9d-9797-bfe21d492b56","Type":"ContainerDied","Data":"a73932a5dd693825cd1eefbfe7abb27633475e0ed405227562d3083526e967bf"} Jul 06 16:01:28.336514 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:28.336490 2561 scope.go:117] "RemoveContainer" containerID="a73932a5dd693825cd1eefbfe7abb27633475e0ed405227562d3083526e967bf" Jul 06 16:01:28.337459 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:28.337434 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:01:28.337550 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:28.337511 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:01:28.337603 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:28.337577 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:01:29.343582 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:29.343518 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" event={"ID":"38cd3ea0-a846-4a9d-9797-bfe21d492b56","Type":"ContainerStarted","Data":"033a727cb91372c62b2828713aad471e4a19a7c125e854d8770c0a46b56699b7"} Jul 06 16:01:29.763578 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:29.763542 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:29.764805 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:29.764777 2561 patch_prober.go:28] interesting pod/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 06 16:01:29.764942 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:29.764839 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" podUID="38cd3ea0-a846-4a9d-9797-bfe21d492b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 06 16:01:30.763655 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:30.763619 2561 patch_prober.go:28] interesting pod/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 06 16:01:30.764026 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:30.763675 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" podUID="38cd3ea0-a846-4a9d-9797-bfe21d492b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 06 16:01:31.055063 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:31.054985 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:31.156421 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:31.156389 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:31.254632 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:31.254599 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:31.763360 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:31.763328 2561 patch_prober.go:28] interesting pod/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 06 16:01:31.763596 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:31.763393 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" podUID="38cd3ea0-a846-4a9d-9797-bfe21d492b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 06 16:01:32.763245 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:32.763208 2561 patch_prober.go:28] interesting pod/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 06 16:01:32.763645 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:32.763267 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" podUID="38cd3ea0-a846-4a9d-9797-bfe21d492b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 06 16:01:33.767677 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:33.767646 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:33.768633 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:33.768613 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:01:37.952450 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:37.952415 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:38.053940 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:38.053897 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:59.291301 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:01:59.291264 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:20.068657 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:20.068579 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:21.105306 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:21.105276 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:21.399253 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:21.399167 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:21.755175 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:21.755148 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:21.856137 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:21.856108 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:21.957073 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:21.957040 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:22.063312 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.063207 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm"] Jul 06 16:02:22.063621 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.063569 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" podUID="b3be9e52-8861-471a-8969-cce362a5161d" containerName="maas-api" containerID="cri-o://8214817e281ed3a988bceab14916c0c191f6c2ce55b0864d2716d8c9937d09f9" gracePeriod=30 Jul 06 16:02:22.086647 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.086625 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:22.228831 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.228784 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" podUID="b3be9e52-8861-471a-8969-cce362a5161d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.61:8443/health\": dial tcp 10.133.0.61:8443: connect: connection refused" Jul 06 16:02:22.328615 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.328556 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:02:22.408545 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.408498 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3be9e52-8861-471a-8969-cce362a5161d-maas-api-tls\") pod \"b3be9e52-8861-471a-8969-cce362a5161d\" (UID: \"b3be9e52-8861-471a-8969-cce362a5161d\") " Jul 06 16:02:22.408728 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.408623 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9bbk\" (UniqueName: \"kubernetes.io/projected/b3be9e52-8861-471a-8969-cce362a5161d-kube-api-access-w9bbk\") pod \"b3be9e52-8861-471a-8969-cce362a5161d\" (UID: \"b3be9e52-8861-471a-8969-cce362a5161d\") " Jul 06 16:02:22.408728 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.408691 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3be9e52-8861-471a-8969-cce362a5161d-openshift-service-ca\") pod \"b3be9e52-8861-471a-8969-cce362a5161d\" (UID: \"b3be9e52-8861-471a-8969-cce362a5161d\") " Jul 06 16:02:22.409032 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.408995 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3be9e52-8861-471a-8969-cce362a5161d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b3be9e52-8861-471a-8969-cce362a5161d" (UID: "b3be9e52-8861-471a-8969-cce362a5161d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:02:22.410877 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.410853 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3be9e52-8861-471a-8969-cce362a5161d-kube-api-access-w9bbk" (OuterVolumeSpecName: "kube-api-access-w9bbk") pod "b3be9e52-8861-471a-8969-cce362a5161d" (UID: "b3be9e52-8861-471a-8969-cce362a5161d"). InnerVolumeSpecName "kube-api-access-w9bbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:02:22.411101 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.411085 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3be9e52-8861-471a-8969-cce362a5161d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b3be9e52-8861-471a-8969-cce362a5161d" (UID: "b3be9e52-8861-471a-8969-cce362a5161d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:02:22.509811 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.509772 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9bbk\" (UniqueName: \"kubernetes.io/projected/b3be9e52-8861-471a-8969-cce362a5161d-kube-api-access-w9bbk\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:02:22.509811 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.509798 2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3be9e52-8861-471a-8969-cce362a5161d-openshift-service-ca\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:02:22.509811 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.509809 2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3be9e52-8861-471a-8969-cce362a5161d-maas-api-tls\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:02:22.627572 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.627475 2561 generic.go:356] "Generic (PLEG): container finished" podID="b3be9e52-8861-471a-8969-cce362a5161d" containerID="8214817e281ed3a988bceab14916c0c191f6c2ce55b0864d2716d8c9937d09f9" exitCode=0 Jul 06 16:02:22.627572 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.627540 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" Jul 06 16:02:22.627744 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.627564 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" event={"ID":"b3be9e52-8861-471a-8969-cce362a5161d","Type":"ContainerDied","Data":"8214817e281ed3a988bceab14916c0c191f6c2ce55b0864d2716d8c9937d09f9"} Jul 06 16:02:22.627744 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.627626 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm" event={"ID":"b3be9e52-8861-471a-8969-cce362a5161d","Type":"ContainerDied","Data":"bd9d76f4c31cc3bfff84c2c19ecddea6e6e17312cb1f587203a7041de30a017d"} Jul 06 16:02:22.627744 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.627653 2561 scope.go:117] "RemoveContainer" containerID="8214817e281ed3a988bceab14916c0c191f6c2ce55b0864d2716d8c9937d09f9" Jul 06 16:02:22.637518 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.637500 2561 scope.go:117] "RemoveContainer" containerID="8214817e281ed3a988bceab14916c0c191f6c2ce55b0864d2716d8c9937d09f9" Jul 06 16:02:22.637825 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:02:22.637808 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8214817e281ed3a988bceab14916c0c191f6c2ce55b0864d2716d8c9937d09f9\": container with ID starting with 8214817e281ed3a988bceab14916c0c191f6c2ce55b0864d2716d8c9937d09f9 not found: ID does not exist" containerID="8214817e281ed3a988bceab14916c0c191f6c2ce55b0864d2716d8c9937d09f9" Jul 06 16:02:22.637873 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.637833 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8214817e281ed3a988bceab14916c0c191f6c2ce55b0864d2716d8c9937d09f9"} err="failed to get container status \"8214817e281ed3a988bceab14916c0c191f6c2ce55b0864d2716d8c9937d09f9\": rpc error: code = NotFound desc = could not find container \"8214817e281ed3a988bceab14916c0c191f6c2ce55b0864d2716d8c9937d09f9\": container with ID starting with 8214817e281ed3a988bceab14916c0c191f6c2ce55b0864d2716d8c9937d09f9 not found: ID does not exist" Jul 06 16:02:22.652071 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.652050 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm"] Jul 06 16:02:22.656488 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.656467 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-cff200-547ff576f5-9vbbm"] Jul 06 16:02:22.800366 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:22.800337 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3be9e52-8861-471a-8969-cce362a5161d" path="/var/lib/kubelet/pods/b3be9e52-8861-471a-8969-cce362a5161d/volumes" Jul 06 16:02:26.834373 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:26.834342 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:27.901148 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:02:27.901111 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:05.189378 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:05.189342 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:05.428906 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:05.428873 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb"] Jul 06 16:03:05.429131 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:05.429097 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" podUID="de13cf94-1748-46b8-bd3e-55a45b65c56e" containerName="istio-proxy" containerID="cri-o://f0f3488de113b82b3c478a26b89722e250e50ea0835248b8d293dea4ff9afcfb" gracePeriod=30 Jul 06 16:03:05.445701 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:05.445631 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:05.690582 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:05.690551 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:05.887772 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:05.887741 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:06.282852 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:06.282824 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:06.398052 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:06.398021 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:06.495477 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:06.495441 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:06.729538 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:06.729499 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:10.694547 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.694459 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:03:10.761706 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.761673 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-envoy\") pod \"de13cf94-1748-46b8-bd3e-55a45b65c56e\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " Jul 06 16:03:10.761837 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.761755 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-workload-socket\") pod \"de13cf94-1748-46b8-bd3e-55a45b65c56e\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " Jul 06 16:03:10.761837 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.761776 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-credential-socket\") pod \"de13cf94-1748-46b8-bd3e-55a45b65c56e\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " Jul 06 16:03:10.761837 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.761794 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-workload-certs\") pod \"de13cf94-1748-46b8-bd3e-55a45b65c56e\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " Jul 06 16:03:10.761837 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.761818 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-token\") pod \"de13cf94-1748-46b8-bd3e-55a45b65c56e\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " Jul 06 16:03:10.763004 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.762042 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "de13cf94-1748-46b8-bd3e-55a45b65c56e" (UID: "de13cf94-1748-46b8-bd3e-55a45b65c56e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:10.763004 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.762110 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-data\") pod \"de13cf94-1748-46b8-bd3e-55a45b65c56e\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " Jul 06 16:03:10.763004 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.762157 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de13cf94-1748-46b8-bd3e-55a45b65c56e-istiod-ca-cert\") pod \"de13cf94-1748-46b8-bd3e-55a45b65c56e\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " Jul 06 16:03:10.763004 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.762208 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-podinfo\") pod \"de13cf94-1748-46b8-bd3e-55a45b65c56e\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " Jul 06 16:03:10.763004 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.762231 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "de13cf94-1748-46b8-bd3e-55a45b65c56e" (UID: "de13cf94-1748-46b8-bd3e-55a45b65c56e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:10.763004 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.762272 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6pcd\" (UniqueName: \"kubernetes.io/projected/de13cf94-1748-46b8-bd3e-55a45b65c56e-kube-api-access-p6pcd\") pod \"de13cf94-1748-46b8-bd3e-55a45b65c56e\" (UID: \"de13cf94-1748-46b8-bd3e-55a45b65c56e\") " Jul 06 16:03:10.763004 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.762294 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "de13cf94-1748-46b8-bd3e-55a45b65c56e" (UID: "de13cf94-1748-46b8-bd3e-55a45b65c56e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:10.763004 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.762591 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de13cf94-1748-46b8-bd3e-55a45b65c56e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "de13cf94-1748-46b8-bd3e-55a45b65c56e" (UID: "de13cf94-1748-46b8-bd3e-55a45b65c56e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:03:10.763004 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.762720 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-workload-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:10.763004 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.762737 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-credential-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:10.763004 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.762750 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-workload-certs\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:10.763004 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.762763 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de13cf94-1748-46b8-bd3e-55a45b65c56e-istiod-ca-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:10.763670 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.763284 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-data" (OuterVolumeSpecName: "istio-data") pod "de13cf94-1748-46b8-bd3e-55a45b65c56e" (UID: "de13cf94-1748-46b8-bd3e-55a45b65c56e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:10.764764 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.764737 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "de13cf94-1748-46b8-bd3e-55a45b65c56e" (UID: "de13cf94-1748-46b8-bd3e-55a45b65c56e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:10.764930 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.764911 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "de13cf94-1748-46b8-bd3e-55a45b65c56e" (UID: "de13cf94-1748-46b8-bd3e-55a45b65c56e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:03:10.765228 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.765197 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de13cf94-1748-46b8-bd3e-55a45b65c56e-kube-api-access-p6pcd" (OuterVolumeSpecName: "kube-api-access-p6pcd") pod "de13cf94-1748-46b8-bd3e-55a45b65c56e" (UID: "de13cf94-1748-46b8-bd3e-55a45b65c56e"). InnerVolumeSpecName "kube-api-access-p6pcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:03:10.765753 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.765734 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-token" (OuterVolumeSpecName: "istio-token") pod "de13cf94-1748-46b8-bd3e-55a45b65c56e" (UID: "de13cf94-1748-46b8-bd3e-55a45b65c56e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:03:10.863425 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.863382 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-token\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:10.863425 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.863420 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-data\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:10.863648 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.863434 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-podinfo\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:10.863648 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.863447 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6pcd\" (UniqueName: \"kubernetes.io/projected/de13cf94-1748-46b8-bd3e-55a45b65c56e-kube-api-access-p6pcd\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:10.863648 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.863459 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de13cf94-1748-46b8-bd3e-55a45b65c56e-istio-envoy\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:10.881176 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.881152 2561 generic.go:356] "Generic (PLEG): container finished" podID="de13cf94-1748-46b8-bd3e-55a45b65c56e" containerID="f0f3488de113b82b3c478a26b89722e250e50ea0835248b8d293dea4ff9afcfb" exitCode=0 Jul 06 16:03:10.881290 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.881192 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" Jul 06 16:03:10.881290 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.881216 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" event={"ID":"de13cf94-1748-46b8-bd3e-55a45b65c56e","Type":"ContainerDied","Data":"f0f3488de113b82b3c478a26b89722e250e50ea0835248b8d293dea4ff9afcfb"} Jul 06 16:03:10.881290 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.881261 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb" event={"ID":"de13cf94-1748-46b8-bd3e-55a45b65c56e","Type":"ContainerDied","Data":"a8ed80f23e31d4767ee2698d47c86a865e73fa7911be9cd3298b29ce9db5edf3"} Jul 06 16:03:10.881290 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.881282 2561 scope.go:117] "RemoveContainer" containerID="f0f3488de113b82b3c478a26b89722e250e50ea0835248b8d293dea4ff9afcfb" Jul 06 16:03:10.890720 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.890703 2561 scope.go:117] "RemoveContainer" containerID="fb1f8b2008b5fe78273580cfcc0407be94a54b75b2c7eded77291644b18e3aac" Jul 06 16:03:10.898589 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.898574 2561 scope.go:117] "RemoveContainer" containerID="f0f3488de113b82b3c478a26b89722e250e50ea0835248b8d293dea4ff9afcfb" Jul 06 16:03:10.898845 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:03:10.898826 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f3488de113b82b3c478a26b89722e250e50ea0835248b8d293dea4ff9afcfb\": container with ID starting with f0f3488de113b82b3c478a26b89722e250e50ea0835248b8d293dea4ff9afcfb not found: ID does not exist" containerID="f0f3488de113b82b3c478a26b89722e250e50ea0835248b8d293dea4ff9afcfb" Jul 06 16:03:10.898906 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.898852 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f3488de113b82b3c478a26b89722e250e50ea0835248b8d293dea4ff9afcfb"} err="failed to get container status \"f0f3488de113b82b3c478a26b89722e250e50ea0835248b8d293dea4ff9afcfb\": rpc error: code = NotFound desc = could not find container \"f0f3488de113b82b3c478a26b89722e250e50ea0835248b8d293dea4ff9afcfb\": container with ID starting with f0f3488de113b82b3c478a26b89722e250e50ea0835248b8d293dea4ff9afcfb not found: ID does not exist" Jul 06 16:03:10.898906 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.898867 2561 scope.go:117] "RemoveContainer" containerID="fb1f8b2008b5fe78273580cfcc0407be94a54b75b2c7eded77291644b18e3aac" Jul 06 16:03:10.899109 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:03:10.899092 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1f8b2008b5fe78273580cfcc0407be94a54b75b2c7eded77291644b18e3aac\": container with ID starting with fb1f8b2008b5fe78273580cfcc0407be94a54b75b2c7eded77291644b18e3aac not found: ID does not exist" containerID="fb1f8b2008b5fe78273580cfcc0407be94a54b75b2c7eded77291644b18e3aac" Jul 06 16:03:10.899149 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.899113 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1f8b2008b5fe78273580cfcc0407be94a54b75b2c7eded77291644b18e3aac"} err="failed to get container status \"fb1f8b2008b5fe78273580cfcc0407be94a54b75b2c7eded77291644b18e3aac\": rpc error: code = NotFound desc = could not find container \"fb1f8b2008b5fe78273580cfcc0407be94a54b75b2c7eded77291644b18e3aac\": container with ID starting with fb1f8b2008b5fe78273580cfcc0407be94a54b75b2c7eded77291644b18e3aac not found: ID does not exist" Jul 06 16:03:10.917538 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.917505 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb"] Jul 06 16:03:10.920700 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:10.920679 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-cff200-openshift-default-6b964cd767-4drfb"] Jul 06 16:03:12.800821 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:12.800788 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de13cf94-1748-46b8-bd3e-55a45b65c56e" path="/var/lib/kubelet/pods/de13cf94-1748-46b8-bd3e-55a45b65c56e/volumes" Jul 06 16:03:13.837676 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:13.837643 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:13.936323 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:13.936290 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:14.210687 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:14.210606 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn"] Jul 06 16:03:14.210836 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:14.210807 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" podUID="38cd3ea0-a846-4a9d-9797-bfe21d492b56" containerName="istio-proxy" containerID="cri-o://033a727cb91372c62b2828713aad471e4a19a7c125e854d8770c0a46b56699b7" gracePeriod=30 Jul 06 16:03:14.250170 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:14.250133 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:14.441830 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:14.441795 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:14.793794 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:14.793756 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:15.155668 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.155589 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4"] Jul 06 16:03:15.156019 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.156007 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de13cf94-1748-46b8-bd3e-55a45b65c56e" containerName="istio-proxy" Jul 06 16:03:15.156064 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.156021 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="de13cf94-1748-46b8-bd3e-55a45b65c56e" containerName="istio-proxy" Jul 06 16:03:15.156064 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.156038 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3be9e52-8861-471a-8969-cce362a5161d" containerName="maas-api" Jul 06 16:03:15.156064 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.156043 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3be9e52-8861-471a-8969-cce362a5161d" containerName="maas-api" Jul 06 16:03:15.156167 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.156106 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="de13cf94-1748-46b8-bd3e-55a45b65c56e" containerName="istio-proxy" Jul 06 16:03:15.156167 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.156115 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3be9e52-8861-471a-8969-cce362a5161d" containerName="maas-api" Jul 06 16:03:15.160837 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.160819 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.164476 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.164453 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-50b37d-openshift-default-dockercfg-bz2rc\"" Jul 06 16:03:15.176068 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.176049 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:15.181307 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.181285 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4"] Jul 06 16:03:15.303244 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.303209 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-envoy\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.303455 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.303260 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-token\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.303455 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.303289 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdtcr\" (UniqueName: \"kubernetes.io/projected/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-kube-api-access-gdtcr\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.303455 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.303341 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istiod-ca-cert\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.303455 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.303417 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-workload-socket\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.303712 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.303464 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-data\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.303712 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.303535 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-credential-socket\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.303712 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.303582 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-workload-certs\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.303712 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.303639 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-podinfo\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.404096 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.404060 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istiod-ca-cert\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.404096 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.404102 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-workload-socket\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.404342 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.404123 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-data\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.404342 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.404149 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-credential-socket\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.404342 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.404168 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-workload-certs\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.404342 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.404317 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-podinfo\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.404582 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.404405 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-envoy\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.404582 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.404442 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-token\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.404582 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.404467 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdtcr\" (UniqueName: \"kubernetes.io/projected/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-kube-api-access-gdtcr\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.404766 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.404700 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-credential-socket\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.404766 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.404717 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-workload-socket\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.404922 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.404895 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istiod-ca-cert\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.405161 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.405137 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-data\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.405161 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.405151 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-workload-certs\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.407579 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.407450 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-envoy\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.413991 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.413968 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-podinfo\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.420347 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.420324 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-token\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.420501 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.420482 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdtcr\" (UniqueName: \"kubernetes.io/projected/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-kube-api-access-gdtcr\") pod \"e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.471154 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.471128 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:15.635185 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.635151 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:15.656815 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.656795 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4"] Jul 06 16:03:15.660072 ip-10-0-129-225 kubenswrapper[2561]: W0706 16:03:15.660009 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8539d5_29e9_4ff3_aed0_8b9882b7da1e.slice/crio-d4adedb854a6cd3e27bd0f45383e0ac376882b12f0e15b843f8485f58cc9d39c WatchSource:0}: Error finding container d4adedb854a6cd3e27bd0f45383e0ac376882b12f0e15b843f8485f58cc9d39c: Status 404 returned error can't find the container with id d4adedb854a6cd3e27bd0f45383e0ac376882b12f0e15b843f8485f58cc9d39c Jul 06 16:03:15.662165 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.662131 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:03:15.662257 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.662199 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:03:15.662257 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.662229 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:03:15.910662 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.910554 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" event={"ID":"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e","Type":"ContainerStarted","Data":"57c1be8daccf27735f8481877189b635ea47e15a8466c165bb9eafd560b29a97"} Jul 06 16:03:15.910662 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.910629 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" event={"ID":"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e","Type":"ContainerStarted","Data":"d4adedb854a6cd3e27bd0f45383e0ac376882b12f0e15b843f8485f58cc9d39c"} Jul 06 16:03:15.935459 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:15.935407 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" podStartSLOduration=0.935391814 podStartE2EDuration="935.391814ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:03:15.932806811 +0000 UTC m=+2217.565358794" watchObservedRunningTime="2026-07-06 16:03:15.935391814 +0000 UTC m=+2217.567943796" Jul 06 16:03:16.472203 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:16.472173 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:16.477162 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:16.477138 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:16.916418 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:16.916320 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:16.917299 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:16.917283 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:18.014572 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:18.014504 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:18.102748 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:18.102717 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:18.165998 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:18.165972 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:18.281934 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:18.281859 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:18.670939 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:18.670833 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:18.770399 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:18.770361 2561 patch_prober.go:28] interesting pod/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 16:03:18.770593 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:18.770441 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" podUID="38cd3ea0-a846-4a9d-9797-bfe21d492b56" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 16:03:18.771173 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:18.771151 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:19.483092 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.483066 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:03:19.549352 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.549320 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-workload-socket\") pod \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " Jul 06 16:03:19.549518 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.549362 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-token\") pod \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " Jul 06 16:03:19.549518 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.549402 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-workload-certs\") pod \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " Jul 06 16:03:19.549518 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.549425 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-data\") pod \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " Jul 06 16:03:19.549518 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.549457 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsjq7\" (UniqueName: \"kubernetes.io/projected/38cd3ea0-a846-4a9d-9797-bfe21d492b56-kube-api-access-wsjq7\") pod \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " Jul 06 16:03:19.549518 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.549493 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-envoy\") pod \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " Jul 06 16:03:19.549813 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.549518 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-podinfo\") pod \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " Jul 06 16:03:19.549813 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.549596 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istiod-ca-cert\") pod \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " Jul 06 16:03:19.549813 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.549634 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "38cd3ea0-a846-4a9d-9797-bfe21d492b56" (UID: "38cd3ea0-a846-4a9d-9797-bfe21d492b56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:19.549813 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.549664 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-credential-socket\") pod \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\" (UID: \"38cd3ea0-a846-4a9d-9797-bfe21d492b56\") " Jul 06 16:03:19.550014 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.549853 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "38cd3ea0-a846-4a9d-9797-bfe21d492b56" (UID: "38cd3ea0-a846-4a9d-9797-bfe21d492b56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:19.550092 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.550067 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "38cd3ea0-a846-4a9d-9797-bfe21d492b56" (UID: "38cd3ea0-a846-4a9d-9797-bfe21d492b56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:19.550148 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.550109 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "38cd3ea0-a846-4a9d-9797-bfe21d492b56" (UID: "38cd3ea0-a846-4a9d-9797-bfe21d492b56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:03:19.550148 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.550125 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-workload-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:19.550355 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.550150 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-workload-certs\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:19.550797 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.550761 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-data" (OuterVolumeSpecName: "istio-data") pod "38cd3ea0-a846-4a9d-9797-bfe21d492b56" (UID: "38cd3ea0-a846-4a9d-9797-bfe21d492b56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:19.551906 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.551873 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "38cd3ea0-a846-4a9d-9797-bfe21d492b56" (UID: "38cd3ea0-a846-4a9d-9797-bfe21d492b56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:03:19.552030 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.552012 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-token" (OuterVolumeSpecName: "istio-token") pod "38cd3ea0-a846-4a9d-9797-bfe21d492b56" (UID: "38cd3ea0-a846-4a9d-9797-bfe21d492b56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:03:19.552119 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.552096 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38cd3ea0-a846-4a9d-9797-bfe21d492b56-kube-api-access-wsjq7" (OuterVolumeSpecName: "kube-api-access-wsjq7") pod "38cd3ea0-a846-4a9d-9797-bfe21d492b56" (UID: "38cd3ea0-a846-4a9d-9797-bfe21d492b56"). InnerVolumeSpecName "kube-api-access-wsjq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:03:19.552164 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.552125 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "38cd3ea0-a846-4a9d-9797-bfe21d492b56" (UID: "38cd3ea0-a846-4a9d-9797-bfe21d492b56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:19.651487 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.651415 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-token\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:19.651487 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.651439 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-data\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:19.651487 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.651449 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsjq7\" (UniqueName: \"kubernetes.io/projected/38cd3ea0-a846-4a9d-9797-bfe21d492b56-kube-api-access-wsjq7\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:19.651487 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.651458 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-envoy\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:19.651487 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.651467 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istio-podinfo\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:19.651487 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.651474 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38cd3ea0-a846-4a9d-9797-bfe21d492b56-istiod-ca-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:19.651487 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.651481 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38cd3ea0-a846-4a9d-9797-bfe21d492b56-credential-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:19.934126 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.934044 2561 generic.go:356] "Generic (PLEG): container finished" podID="38cd3ea0-a846-4a9d-9797-bfe21d492b56" containerID="033a727cb91372c62b2828713aad471e4a19a7c125e854d8770c0a46b56699b7" exitCode=0 Jul 06 16:03:19.934126 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.934093 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" Jul 06 16:03:19.934353 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.934139 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" event={"ID":"38cd3ea0-a846-4a9d-9797-bfe21d492b56","Type":"ContainerDied","Data":"033a727cb91372c62b2828713aad471e4a19a7c125e854d8770c0a46b56699b7"} Jul 06 16:03:19.934353 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.934203 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn" event={"ID":"38cd3ea0-a846-4a9d-9797-bfe21d492b56","Type":"ContainerDied","Data":"09fae4df9535b5a9cf69fcd1b75d1d0eade31351362c5b553148b38b83eac636"} Jul 06 16:03:19.934353 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.934237 2561 scope.go:117] "RemoveContainer" containerID="033a727cb91372c62b2828713aad471e4a19a7c125e854d8770c0a46b56699b7" Jul 06 16:03:19.944384 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.944371 2561 scope.go:117] "RemoveContainer" containerID="a73932a5dd693825cd1eefbfe7abb27633475e0ed405227562d3083526e967bf" Jul 06 16:03:19.953315 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.953299 2561 scope.go:117] "RemoveContainer" containerID="033a727cb91372c62b2828713aad471e4a19a7c125e854d8770c0a46b56699b7" Jul 06 16:03:19.953614 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:03:19.953595 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033a727cb91372c62b2828713aad471e4a19a7c125e854d8770c0a46b56699b7\": container with ID starting with 033a727cb91372c62b2828713aad471e4a19a7c125e854d8770c0a46b56699b7 not found: ID does not exist" containerID="033a727cb91372c62b2828713aad471e4a19a7c125e854d8770c0a46b56699b7" Jul 06 16:03:19.953669 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.953623 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033a727cb91372c62b2828713aad471e4a19a7c125e854d8770c0a46b56699b7"} err="failed to get container status \"033a727cb91372c62b2828713aad471e4a19a7c125e854d8770c0a46b56699b7\": rpc error: code = NotFound desc = could not find container \"033a727cb91372c62b2828713aad471e4a19a7c125e854d8770c0a46b56699b7\": container with ID starting with 033a727cb91372c62b2828713aad471e4a19a7c125e854d8770c0a46b56699b7 not found: ID does not exist" Jul 06 16:03:19.953669 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.953642 2561 scope.go:117] "RemoveContainer" containerID="a73932a5dd693825cd1eefbfe7abb27633475e0ed405227562d3083526e967bf" Jul 06 16:03:19.953925 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:03:19.953902 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73932a5dd693825cd1eefbfe7abb27633475e0ed405227562d3083526e967bf\": container with ID starting with a73932a5dd693825cd1eefbfe7abb27633475e0ed405227562d3083526e967bf not found: ID does not exist" containerID="a73932a5dd693825cd1eefbfe7abb27633475e0ed405227562d3083526e967bf" Jul 06 16:03:19.954009 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.953930 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73932a5dd693825cd1eefbfe7abb27633475e0ed405227562d3083526e967bf"} err="failed to get container status \"a73932a5dd693825cd1eefbfe7abb27633475e0ed405227562d3083526e967bf\": rpc error: code = NotFound desc = could not find container \"a73932a5dd693825cd1eefbfe7abb27633475e0ed405227562d3083526e967bf\": container with ID starting with a73932a5dd693825cd1eefbfe7abb27633475e0ed405227562d3083526e967bf not found: ID does not exist" Jul 06 16:03:19.961688 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.961667 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn"] Jul 06 16:03:19.967895 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:19.967876 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-57aaf3-openshift-default-5d9bbbc9df-csgqn"] Jul 06 16:03:20.799823 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:20.799792 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38cd3ea0-a846-4a9d-9797-bfe21d492b56" path="/var/lib/kubelet/pods/38cd3ea0-a846-4a9d-9797-bfe21d492b56/volumes" Jul 06 16:03:22.506281 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:22.506248 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:22.963353 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:22.963276 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:23.248523 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.248494 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh"] Jul 06 16:03:23.249026 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.248998 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38cd3ea0-a846-4a9d-9797-bfe21d492b56" containerName="istio-proxy" Jul 06 16:03:23.249026 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.249020 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="38cd3ea0-a846-4a9d-9797-bfe21d492b56" containerName="istio-proxy" Jul 06 16:03:23.249147 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.249041 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38cd3ea0-a846-4a9d-9797-bfe21d492b56" containerName="istio-proxy" Jul 06 16:03:23.249147 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.249049 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="38cd3ea0-a846-4a9d-9797-bfe21d492b56" containerName="istio-proxy" Jul 06 16:03:23.249147 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.249061 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de13cf94-1748-46b8-bd3e-55a45b65c56e" containerName="istio-proxy" Jul 06 16:03:23.249147 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.249067 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="de13cf94-1748-46b8-bd3e-55a45b65c56e" containerName="istio-proxy" Jul 06 16:03:23.249147 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.249148 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="38cd3ea0-a846-4a9d-9797-bfe21d492b56" containerName="istio-proxy" Jul 06 16:03:23.249382 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.249155 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="de13cf94-1748-46b8-bd3e-55a45b65c56e" containerName="istio-proxy" Jul 06 16:03:23.252377 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.252354 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.255683 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.255660 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3c2fc9-openshift-default-dockercfg-kxbmx\"" Jul 06 16:03:23.269787 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.269767 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh"] Jul 06 16:03:23.286477 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.286450 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:23.382945 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.382914 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:23.391158 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.391135 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-credential-socket\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.391289 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.391174 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-workload-socket\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.391289 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.391193 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istiod-ca-cert\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.391360 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.391300 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nptbt\" (UniqueName: \"kubernetes.io/projected/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-kube-api-access-nptbt\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.391360 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.391341 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-token\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.391428 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.391384 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-envoy\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.391465 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.391427 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-data\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.391502 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.391460 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-podinfo\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.391558 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.391515 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-workload-certs\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.493118 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.493082 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-podinfo\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.493118 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.493120 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-workload-certs\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.493358 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.493163 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-credential-socket\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.493358 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.493192 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-workload-socket\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.493358 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.493210 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istiod-ca-cert\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.493484 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.493357 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nptbt\" (UniqueName: \"kubernetes.io/projected/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-kube-api-access-nptbt\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.493484 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.493393 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-token\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.493484 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.493450 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-envoy\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.493699 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.493488 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-data\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.493699 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.493673 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-workload-certs\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.493805 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.493704 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-workload-socket\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.493890 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.493866 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-credential-socket\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.493942 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.493908 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-data\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.494183 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.494165 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istiod-ca-cert\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.495922 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.495902 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-podinfo\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.495922 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.495910 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-envoy\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.511668 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.511597 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nptbt\" (UniqueName: \"kubernetes.io/projected/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-kube-api-access-nptbt\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.513389 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.513359 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-token\") pod \"e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.566142 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.566113 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:23.715747 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.715511 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh"] Jul 06 16:03:23.718116 ip-10-0-129-225 kubenswrapper[2561]: W0706 16:03:23.718086 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e5455d_dd10_4ca9_ab17_2322c2a4ae08.slice/crio-27738757c8271a957b89f368d5861ff6512d27b61c004e4b4961d8733b223896 WatchSource:0}: Error finding container 27738757c8271a957b89f368d5861ff6512d27b61c004e4b4961d8733b223896: Status 404 returned error can't find the container with id 27738757c8271a957b89f368d5861ff6512d27b61c004e4b4961d8733b223896 Jul 06 16:03:23.720373 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.720344 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:03:23.720452 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.720409 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:03:23.720452 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.720438 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:03:23.958383 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.958342 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" event={"ID":"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08","Type":"ContainerStarted","Data":"eb48efb60e02b03d5c08b238daad20899e138fe87867d92b638c1da3f8b7a5f1"} Jul 06 16:03:23.958596 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.958406 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" event={"ID":"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08","Type":"ContainerStarted","Data":"27738757c8271a957b89f368d5861ff6512d27b61c004e4b4961d8733b223896"} Jul 06 16:03:23.982456 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:23.982407 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" podStartSLOduration=0.982385994 podStartE2EDuration="982.385994ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:03:23.981434325 +0000 UTC m=+2225.613986310" watchObservedRunningTime="2026-07-06 16:03:23.982385994 +0000 UTC m=+2225.614937980" Jul 06 16:03:24.568261 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:24.568217 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:24.577973 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:24.577943 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:24.964801 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:24.964708 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:24.965626 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:24.965603 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:26.061540 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:26.061487 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:26.146129 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:26.146093 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:26.233270 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:26.233222 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:26.412731 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:26.412614 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:27.080378 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:27.080348 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:27.190020 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:27.189994 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:30.531991 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:30.531948 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:31.918446 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:31.918030 2561 patch_prober.go:28] interesting pod/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 06 16:03:31.918446 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:31.918104 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" podUID="ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 06 16:03:32.002377 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:32.002347 2561 generic.go:356] "Generic (PLEG): container finished" podID="ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" containerID="57c1be8daccf27735f8481877189b635ea47e15a8466c165bb9eafd560b29a97" exitCode=0 Jul 06 16:03:32.002563 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:32.002402 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" event={"ID":"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e","Type":"ContainerDied","Data":"57c1be8daccf27735f8481877189b635ea47e15a8466c165bb9eafd560b29a97"} Jul 06 16:03:32.002822 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:32.002806 2561 scope.go:117] "RemoveContainer" containerID="57c1be8daccf27735f8481877189b635ea47e15a8466c165bb9eafd560b29a97" Jul 06 16:03:32.003821 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:32.003787 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:03:32.003942 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:32.003862 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:03:32.003942 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:32.003892 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:03:32.207129 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:32.207091 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:32.308891 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:32.308759 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:33.011544 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:33.010551 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" event={"ID":"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e","Type":"ContainerStarted","Data":"5205686286ce81e3d1f95b5830760223e5fa4ccc8d1927ccafe84e963d9742b8"} Jul 06 16:03:33.472226 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:33.472192 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:33.472651 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:33.472626 2561 patch_prober.go:28] interesting pod/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 06 16:03:33.472772 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:33.472684 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" podUID="ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 06 16:03:34.471843 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:34.471806 2561 patch_prober.go:28] interesting pod/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 06 16:03:34.472220 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:34.471877 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" podUID="ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 06 16:03:35.471763 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:35.471727 2561 patch_prober.go:28] interesting pod/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 06 16:03:35.471951 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:35.471787 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" podUID="ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 06 16:03:36.472560 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:36.472505 2561 patch_prober.go:28] interesting pod/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 06 16:03:36.472917 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:36.472596 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" podUID="ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 06 16:03:37.476102 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:37.476058 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:37.476512 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:37.476444 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:37.477205 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:37.477188 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:38.798012 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:38.797976 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:39.156439 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:39.156406 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:39.273259 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:39.273232 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:39.375401 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:39.375279 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:39.552017 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:39.551988 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:39.966191 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:39.966111 2561 patch_prober.go:28] interesting pod/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 06 16:03:39.966191 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:39.966179 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" podUID="a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 06 16:03:40.065935 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:40.065903 2561 generic.go:356] "Generic (PLEG): container finished" podID="a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" containerID="eb48efb60e02b03d5c08b238daad20899e138fe87867d92b638c1da3f8b7a5f1" exitCode=0 Jul 06 16:03:40.066109 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:40.065982 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" event={"ID":"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08","Type":"ContainerDied","Data":"eb48efb60e02b03d5c08b238daad20899e138fe87867d92b638c1da3f8b7a5f1"} Jul 06 16:03:40.066397 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:40.066380 2561 scope.go:117] "RemoveContainer" containerID="eb48efb60e02b03d5c08b238daad20899e138fe87867d92b638c1da3f8b7a5f1" Jul 06 16:03:40.067499 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:40.067473 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:03:40.067595 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:40.067577 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:03:40.067637 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:40.067624 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:03:41.073552 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:41.073498 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" event={"ID":"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08","Type":"ContainerStarted","Data":"37ba582b922dc7a0902b7eb84960f6474c4a9a62881e740de928b49caf7c3ada"} Jul 06 16:03:41.566762 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:41.566729 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:41.566941 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:41.566919 2561 patch_prober.go:28] interesting pod/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 06 16:03:41.566992 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:41.566964 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" podUID="a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 06 16:03:42.566458 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:42.566421 2561 patch_prober.go:28] interesting pod/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 06 16:03:42.566899 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:42.566473 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" podUID="a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 06 16:03:43.567359 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:43.567318 2561 patch_prober.go:28] interesting pod/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 06 16:03:43.567838 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:43.567379 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" podUID="a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 06 16:03:44.567291 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:44.567256 2561 patch_prober.go:28] interesting pod/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 06 16:03:44.567464 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:44.567315 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" podUID="a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 06 16:03:45.500270 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:45.500238 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:45.570680 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:45.570650 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:45.571139 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:45.571114 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:45.571831 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:45.571813 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:45.605412 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:45.605378 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:45.767360 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:45.767277 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4"] Jul 06 16:03:45.767625 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:45.767600 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" podUID="ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" containerName="istio-proxy" containerID="cri-o://5205686286ce81e3d1f95b5830760223e5fa4ccc8d1927ccafe84e963d9742b8" gracePeriod=30 Jul 06 16:03:45.796346 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:45.796321 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:46.020464 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:46.020372 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:46.207383 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:46.207349 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:46.590340 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:46.590296 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:46.690008 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:46.689981 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:46.790966 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:46.790935 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:46.950242 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:46.950159 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:51.029444 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.029389 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:51.131701 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.131661 2561 generic.go:356] "Generic (PLEG): container finished" podID="ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" containerID="5205686286ce81e3d1f95b5830760223e5fa4ccc8d1927ccafe84e963d9742b8" exitCode=0 Jul 06 16:03:51.131891 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.131718 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" Jul 06 16:03:51.131891 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.131746 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" event={"ID":"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e","Type":"ContainerDied","Data":"5205686286ce81e3d1f95b5830760223e5fa4ccc8d1927ccafe84e963d9742b8"} Jul 06 16:03:51.131891 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.131815 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4" event={"ID":"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e","Type":"ContainerDied","Data":"d4adedb854a6cd3e27bd0f45383e0ac376882b12f0e15b843f8485f58cc9d39c"} Jul 06 16:03:51.131891 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.131849 2561 scope.go:117] "RemoveContainer" containerID="5205686286ce81e3d1f95b5830760223e5fa4ccc8d1927ccafe84e963d9742b8" Jul 06 16:03:51.143100 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.143082 2561 scope.go:117] "RemoveContainer" containerID="57c1be8daccf27735f8481877189b635ea47e15a8466c165bb9eafd560b29a97" Jul 06 16:03:51.151512 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.151492 2561 scope.go:117] "RemoveContainer" containerID="5205686286ce81e3d1f95b5830760223e5fa4ccc8d1927ccafe84e963d9742b8" Jul 06 16:03:51.151880 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:03:51.151850 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5205686286ce81e3d1f95b5830760223e5fa4ccc8d1927ccafe84e963d9742b8\": container with ID starting with 5205686286ce81e3d1f95b5830760223e5fa4ccc8d1927ccafe84e963d9742b8 not found: ID does not exist" containerID="5205686286ce81e3d1f95b5830760223e5fa4ccc8d1927ccafe84e963d9742b8" Jul 06 16:03:51.151929 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.151889 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5205686286ce81e3d1f95b5830760223e5fa4ccc8d1927ccafe84e963d9742b8"} err="failed to get container status \"5205686286ce81e3d1f95b5830760223e5fa4ccc8d1927ccafe84e963d9742b8\": rpc error: code = NotFound desc = could not find container \"5205686286ce81e3d1f95b5830760223e5fa4ccc8d1927ccafe84e963d9742b8\": container with ID starting with 5205686286ce81e3d1f95b5830760223e5fa4ccc8d1927ccafe84e963d9742b8 not found: ID does not exist" Jul 06 16:03:51.151929 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.151909 2561 scope.go:117] "RemoveContainer" containerID="57c1be8daccf27735f8481877189b635ea47e15a8466c165bb9eafd560b29a97" Jul 06 16:03:51.152169 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:03:51.152151 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c1be8daccf27735f8481877189b635ea47e15a8466c165bb9eafd560b29a97\": container with ID starting with 57c1be8daccf27735f8481877189b635ea47e15a8466c165bb9eafd560b29a97 not found: ID does not exist" containerID="57c1be8daccf27735f8481877189b635ea47e15a8466c165bb9eafd560b29a97" Jul 06 16:03:51.152211 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.152176 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c1be8daccf27735f8481877189b635ea47e15a8466c165bb9eafd560b29a97"} err="failed to get container status \"57c1be8daccf27735f8481877189b635ea47e15a8466c165bb9eafd560b29a97\": rpc error: code = NotFound desc = could not find container \"57c1be8daccf27735f8481877189b635ea47e15a8466c165bb9eafd560b29a97\": container with ID starting with 57c1be8daccf27735f8481877189b635ea47e15a8466c165bb9eafd560b29a97 not found: ID does not exist" Jul 06 16:03:51.199909 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.199884 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-workload-certs\") pod \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " Jul 06 16:03:51.200045 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.199959 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-workload-socket\") pod \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " Jul 06 16:03:51.200045 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.199996 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-envoy\") pod \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " Jul 06 16:03:51.200045 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.200028 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdtcr\" (UniqueName: \"kubernetes.io/projected/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-kube-api-access-gdtcr\") pod \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " Jul 06 16:03:51.200211 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.200103 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istiod-ca-cert\") pod \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " Jul 06 16:03:51.200211 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.200116 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" (UID: "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:51.200211 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.200143 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-token\") pod \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " Jul 06 16:03:51.200211 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.200184 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-credential-socket\") pod \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " Jul 06 16:03:51.200414 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.200237 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-data\") pod \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " Jul 06 16:03:51.200414 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.200255 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" (UID: "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:51.200414 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.200264 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-podinfo\") pod \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\" (UID: \"ae8539d5-29e9-4ff3-aed0-8b9882b7da1e\") " Jul 06 16:03:51.200606 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.200451 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" (UID: "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:51.200606 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.200467 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" (UID: "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:03:51.200822 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.200804 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-workload-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:51.200900 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.200825 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istiod-ca-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:51.200900 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.200836 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-credential-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:51.200900 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.200844 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-workload-certs\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:51.201616 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.201589 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-data" (OuterVolumeSpecName: "istio-data") pod "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" (UID: "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:51.202312 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.202289 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" (UID: "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:51.202642 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.202622 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" (UID: "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:03:51.202803 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.202789 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-token" (OuterVolumeSpecName: "istio-token") pod "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" (UID: "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:03:51.202852 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.202804 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-kube-api-access-gdtcr" (OuterVolumeSpecName: "kube-api-access-gdtcr") pod "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" (UID: "ae8539d5-29e9-4ff3-aed0-8b9882b7da1e"). InnerVolumeSpecName "kube-api-access-gdtcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:03:51.302356 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.302264 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-envoy\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:51.302356 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.302297 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdtcr\" (UniqueName: \"kubernetes.io/projected/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-kube-api-access-gdtcr\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:51.302356 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.302310 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-token\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:51.302356 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.302321 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-data\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:51.302356 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.302333 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e-istio-podinfo\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:51.456727 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.456698 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4"] Jul 06 16:03:51.460689 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:51.460665 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-50b37d-openshift-default-67f5799885-6v6r4"] Jul 06 16:03:52.800161 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:52.800115 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" path="/var/lib/kubelet/pods/ae8539d5-29e9-4ff3-aed0-8b9882b7da1e/volumes" Jul 06 16:03:53.018955 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:53.018915 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:53.116769 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:53.116684 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:53.264156 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:53.264125 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh"] Jul 06 16:03:53.264342 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:53.264316 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" podUID="a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" containerName="istio-proxy" containerID="cri-o://37ba582b922dc7a0902b7eb84960f6474c4a9a62881e740de928b49caf7c3ada" gracePeriod=30 Jul 06 16:03:53.292811 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:53.292786 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:53.526094 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:53.526061 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:53.858268 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:53.858188 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:54.145492 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.145416 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6"] Jul 06 16:03:54.146034 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.146018 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" containerName="istio-proxy" Jul 06 16:03:54.146089 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.146038 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" containerName="istio-proxy" Jul 06 16:03:54.146089 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.146076 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" containerName="istio-proxy" Jul 06 16:03:54.146089 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.146085 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" containerName="istio-proxy" Jul 06 16:03:54.146216 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.146185 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" containerName="istio-proxy" Jul 06 16:03:54.146216 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.146198 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="38cd3ea0-a846-4a9d-9797-bfe21d492b56" containerName="istio-proxy" Jul 06 16:03:54.149857 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.149840 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.152753 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.152731 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-b75322-openshift-default-dockercfg-tqr2l\"" Jul 06 16:03:54.160955 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.160931 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6"] Jul 06 16:03:54.166202 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.166181 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:54.337871 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.337834 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-workload-certs\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.338068 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.337897 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-workload-socket\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.338068 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.337940 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27deb48-2f82-405d-b6c4-13209d22b934-istio-token\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.338068 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.337957 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb2cv\" (UniqueName: \"kubernetes.io/projected/c27deb48-2f82-405d-b6c4-13209d22b934-kube-api-access-zb2cv\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.338068 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.337994 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27deb48-2f82-405d-b6c4-13209d22b934-istiod-ca-cert\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.338068 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.338027 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-istio-data\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.338340 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.338114 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27deb48-2f82-405d-b6c4-13209d22b934-istio-podinfo\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.338340 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.338143 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-credential-socket\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.338340 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.338161 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-istio-envoy\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.416522 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.416446 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:54.439607 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.439578 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27deb48-2f82-405d-b6c4-13209d22b934-istio-podinfo\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.439729 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.439609 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-credential-socket\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.439729 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.439629 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-istio-envoy\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.439729 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.439678 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-workload-certs\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.439851 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.439754 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-workload-socket\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.439851 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.439779 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27deb48-2f82-405d-b6c4-13209d22b934-istio-token\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.439851 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.439802 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb2cv\" (UniqueName: \"kubernetes.io/projected/c27deb48-2f82-405d-b6c4-13209d22b934-kube-api-access-zb2cv\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.439999 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.439832 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27deb48-2f82-405d-b6c4-13209d22b934-istiod-ca-cert\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.439999 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.439890 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-istio-data\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.440101 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.439999 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-credential-socket\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.440390 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.440365 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-workload-socket\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.440486 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.440401 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-workload-certs\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.440486 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.440427 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-istio-data\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.440629 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.440612 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27deb48-2f82-405d-b6c4-13209d22b934-istiod-ca-cert\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.442597 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.442575 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-istio-envoy\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.442916 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.442898 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27deb48-2f82-405d-b6c4-13209d22b934-istio-podinfo\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.449670 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.449646 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27deb48-2f82-405d-b6c4-13209d22b934-istio-token\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.449786 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.449741 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb2cv\" (UniqueName: \"kubernetes.io/projected/c27deb48-2f82-405d-b6c4-13209d22b934-kube-api-access-zb2cv\") pod \"e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.461678 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.461657 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:54.604559 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.604506 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6"] Jul 06 16:03:54.606646 ip-10-0-129-225 kubenswrapper[2561]: W0706 16:03:54.606610 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27deb48_2f82_405d_b6c4_13209d22b934.slice/crio-e6c09559cecbab511fab68bbb8e8e60ca5c07d75c546224e2c09067da384ae9f WatchSource:0}: Error finding container e6c09559cecbab511fab68bbb8e8e60ca5c07d75c546224e2c09067da384ae9f: Status 404 returned error can't find the container with id e6c09559cecbab511fab68bbb8e8e60ca5c07d75c546224e2c09067da384ae9f Jul 06 16:03:54.609765 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.609728 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:03:54.609860 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.609815 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:03:54.609927 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:54.609858 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:03:55.154126 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:55.154092 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" event={"ID":"c27deb48-2f82-405d-b6c4-13209d22b934","Type":"ContainerStarted","Data":"e789b103a11680bd2b79f6c6a68821a906ebdb947ee818b371465112b99766cd"} Jul 06 16:03:55.154617 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:55.154145 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" event={"ID":"c27deb48-2f82-405d-b6c4-13209d22b934","Type":"ContainerStarted","Data":"e6c09559cecbab511fab68bbb8e8e60ca5c07d75c546224e2c09067da384ae9f"} Jul 06 16:03:55.181609 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:55.181557 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" podStartSLOduration=1.181539307 podStartE2EDuration="1.181539307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:03:55.178612982 +0000 UTC m=+2256.811164965" watchObservedRunningTime="2026-07-06 16:03:55.181539307 +0000 UTC m=+2256.814091282" Jul 06 16:03:55.462521 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:55.462435 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:55.467503 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:55.467478 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:56.159988 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:56.159956 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:56.160812 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:56.160793 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:03:58.331138 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.331108 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:58.441253 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.441214 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:58.529385 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.529329 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:58.577424 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.577396 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-podinfo\") pod \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " Jul 06 16:03:58.577573 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.577453 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nptbt\" (UniqueName: \"kubernetes.io/projected/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-kube-api-access-nptbt\") pod \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " Jul 06 16:03:58.577573 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.577484 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-workload-certs\") pod \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " Jul 06 16:03:58.577573 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.577545 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-token\") pod \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " Jul 06 16:03:58.577755 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.577584 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-workload-socket\") pod \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " Jul 06 16:03:58.577755 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.577669 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-data\") pod \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " Jul 06 16:03:58.577755 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.577707 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-credential-socket\") pod \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " Jul 06 16:03:58.577910 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.577767 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-envoy\") pod \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " Jul 06 16:03:58.577910 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.577800 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istiod-ca-cert\") pod \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\" (UID: \"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08\") " Jul 06 16:03:58.577910 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.577849 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" (UID: "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:58.578161 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.578139 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-workload-certs\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:58.578814 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.578655 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" (UID: "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:58.578814 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.578700 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" (UID: "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:03:58.579223 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.579188 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" (UID: "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:58.579375 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.579349 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-data" (OuterVolumeSpecName: "istio-data") pod "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" (UID: "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:58.580118 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.580092 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-token" (OuterVolumeSpecName: "istio-token") pod "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" (UID: "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:03:58.580291 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.580268 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" (UID: "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:03:58.580342 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.580277 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-kube-api-access-nptbt" (OuterVolumeSpecName: "kube-api-access-nptbt") pod "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" (UID: "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08"). InnerVolumeSpecName "kube-api-access-nptbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:03:58.580391 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.580367 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" (UID: "a0e5455d-dd10-4ca9-ab17-2322c2a4ae08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:03:58.678966 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.678939 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-data\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:58.678966 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.678962 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-credential-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:58.678966 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.678972 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-envoy\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:58.679152 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.678981 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istiod-ca-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:58.679152 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.678988 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-podinfo\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:58.679152 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.678997 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nptbt\" (UniqueName: \"kubernetes.io/projected/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-kube-api-access-nptbt\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:58.679152 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.679006 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-istio-token\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:58.679152 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:58.679016 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08-workload-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:03:59.177670 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:59.177587 2561 generic.go:356] "Generic (PLEG): container finished" podID="a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" containerID="37ba582b922dc7a0902b7eb84960f6474c4a9a62881e740de928b49caf7c3ada" exitCode=0 Jul 06 16:03:59.177670 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:59.177640 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" Jul 06 16:03:59.177881 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:59.177670 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" event={"ID":"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08","Type":"ContainerDied","Data":"37ba582b922dc7a0902b7eb84960f6474c4a9a62881e740de928b49caf7c3ada"} Jul 06 16:03:59.177881 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:59.177723 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh" event={"ID":"a0e5455d-dd10-4ca9-ab17-2322c2a4ae08","Type":"ContainerDied","Data":"27738757c8271a957b89f368d5861ff6512d27b61c004e4b4961d8733b223896"} Jul 06 16:03:59.177881 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:59.177746 2561 scope.go:117] "RemoveContainer" containerID="37ba582b922dc7a0902b7eb84960f6474c4a9a62881e740de928b49caf7c3ada" Jul 06 16:03:59.187590 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:59.187569 2561 scope.go:117] "RemoveContainer" containerID="eb48efb60e02b03d5c08b238daad20899e138fe87867d92b638c1da3f8b7a5f1" Jul 06 16:03:59.196056 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:59.196035 2561 scope.go:117] "RemoveContainer" containerID="37ba582b922dc7a0902b7eb84960f6474c4a9a62881e740de928b49caf7c3ada" Jul 06 16:03:59.196307 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:03:59.196285 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ba582b922dc7a0902b7eb84960f6474c4a9a62881e740de928b49caf7c3ada\": container with ID starting with 37ba582b922dc7a0902b7eb84960f6474c4a9a62881e740de928b49caf7c3ada not found: ID does not exist" containerID="37ba582b922dc7a0902b7eb84960f6474c4a9a62881e740de928b49caf7c3ada" Jul 06 16:03:59.196386 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:59.196318 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ba582b922dc7a0902b7eb84960f6474c4a9a62881e740de928b49caf7c3ada"} err="failed to get container status \"37ba582b922dc7a0902b7eb84960f6474c4a9a62881e740de928b49caf7c3ada\": rpc error: code = NotFound desc = could not find container \"37ba582b922dc7a0902b7eb84960f6474c4a9a62881e740de928b49caf7c3ada\": container with ID starting with 37ba582b922dc7a0902b7eb84960f6474c4a9a62881e740de928b49caf7c3ada not found: ID does not exist" Jul 06 16:03:59.196386 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:59.196335 2561 scope.go:117] "RemoveContainer" containerID="eb48efb60e02b03d5c08b238daad20899e138fe87867d92b638c1da3f8b7a5f1" Jul 06 16:03:59.196634 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:03:59.196615 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb48efb60e02b03d5c08b238daad20899e138fe87867d92b638c1da3f8b7a5f1\": container with ID starting with eb48efb60e02b03d5c08b238daad20899e138fe87867d92b638c1da3f8b7a5f1 not found: ID does not exist" containerID="eb48efb60e02b03d5c08b238daad20899e138fe87867d92b638c1da3f8b7a5f1" Jul 06 16:03:59.196702 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:59.196642 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb48efb60e02b03d5c08b238daad20899e138fe87867d92b638c1da3f8b7a5f1"} err="failed to get container status \"eb48efb60e02b03d5c08b238daad20899e138fe87867d92b638c1da3f8b7a5f1\": rpc error: code = NotFound desc = could not find container \"eb48efb60e02b03d5c08b238daad20899e138fe87867d92b638c1da3f8b7a5f1\": container with ID starting with eb48efb60e02b03d5c08b238daad20899e138fe87867d92b638c1da3f8b7a5f1 not found: ID does not exist" Jul 06 16:03:59.198677 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:59.198657 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh"] Jul 06 16:03:59.200713 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:03:59.200696 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3c2fc9-openshift-default-586fd5bbb9-24hxh"] Jul 06 16:04:00.801253 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:00.801217 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" path="/var/lib/kubelet/pods/a0e5455d-dd10-4ca9-ab17-2322c2a4ae08/volumes" Jul 06 16:04:01.708858 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:01.708825 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:02.084302 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.084273 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:02.324737 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.324699 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l"] Jul 06 16:04:02.325247 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.325233 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" containerName="istio-proxy" Jul 06 16:04:02.325305 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.325249 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" containerName="istio-proxy" Jul 06 16:04:02.325305 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.325263 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" containerName="istio-proxy" Jul 06 16:04:02.325305 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.325268 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" containerName="istio-proxy" Jul 06 16:04:02.325396 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.325324 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae8539d5-29e9-4ff3-aed0-8b9882b7da1e" containerName="istio-proxy" Jul 06 16:04:02.325396 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.325337 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" containerName="istio-proxy" Jul 06 16:04:02.325396 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.325342 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e5455d-dd10-4ca9-ab17-2322c2a4ae08" containerName="istio-proxy" Jul 06 16:04:02.329870 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.329851 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.333295 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.333273 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-690f40-openshift-default-dockercfg-c6pz8\"" Jul 06 16:04:02.347150 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.347132 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l"] Jul 06 16:04:02.396401 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.396375 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:02.418045 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.418002 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-istio-envoy\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.418204 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.418054 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-istio-data\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.418204 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.418090 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f409be-d908-4e0a-8488-c8f2a9703619-istio-podinfo\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.418204 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.418138 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9dzk\" (UniqueName: \"kubernetes.io/projected/97f409be-d908-4e0a-8488-c8f2a9703619-kube-api-access-f9dzk\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.418204 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.418184 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-workload-socket\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.418483 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.418236 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f409be-d908-4e0a-8488-c8f2a9703619-istiod-ca-cert\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.418483 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.418261 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f409be-d908-4e0a-8488-c8f2a9703619-istio-token\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.418483 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.418325 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-workload-certs\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.418667 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.418523 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-credential-socket\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.520029 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.519994 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-credential-socket\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.520029 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.520028 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-istio-envoy\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.520248 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.520044 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-istio-data\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.520248 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.520187 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f409be-d908-4e0a-8488-c8f2a9703619-istio-podinfo\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.520340 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.520262 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9dzk\" (UniqueName: \"kubernetes.io/projected/97f409be-d908-4e0a-8488-c8f2a9703619-kube-api-access-f9dzk\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.520340 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.520319 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-workload-socket\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.520438 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.520385 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-istio-data\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.520438 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.520390 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f409be-d908-4e0a-8488-c8f2a9703619-istiod-ca-cert\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.520553 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.520443 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f409be-d908-4e0a-8488-c8f2a9703619-istio-token\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.520553 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.520461 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-credential-socket\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.520553 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.520483 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-workload-certs\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.520815 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.520782 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-workload-certs\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.521015 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.520995 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-workload-socket\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.521120 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.521100 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f409be-d908-4e0a-8488-c8f2a9703619-istiod-ca-cert\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.522616 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.522592 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-istio-envoy\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.522822 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.522801 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f409be-d908-4e0a-8488-c8f2a9703619-istio-podinfo\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.536492 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.536467 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9dzk\" (UniqueName: \"kubernetes.io/projected/97f409be-d908-4e0a-8488-c8f2a9703619-kube-api-access-f9dzk\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.538048 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.538030 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f409be-d908-4e0a-8488-c8f2a9703619-istio-token\") pod \"e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.640365 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.640297 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:02.790969 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.790941 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l"] Jul 06 16:04:02.793097 ip-10-0-129-225 kubenswrapper[2561]: W0706 16:04:02.793070 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f409be_d908_4e0a_8488_c8f2a9703619.slice/crio-93ec7cd8dba49fc49e338ee64e3790f03a2cca8ae4a0ea4fbe8d0d7507aba4d1 WatchSource:0}: Error finding container 93ec7cd8dba49fc49e338ee64e3790f03a2cca8ae4a0ea4fbe8d0d7507aba4d1: Status 404 returned error can't find the container with id 93ec7cd8dba49fc49e338ee64e3790f03a2cca8ae4a0ea4fbe8d0d7507aba4d1 Jul 06 16:04:02.795284 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.795252 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:04:02.795397 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.795327 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:04:02.795397 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:02.795372 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:04:03.137202 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:03.137168 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:03.202519 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:03.202472 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" event={"ID":"97f409be-d908-4e0a-8488-c8f2a9703619","Type":"ContainerStarted","Data":"657f5a0a721a441acc3e5efdc7e0f9ec54ed978a97afd05ce6d7177cf49e3b73"} Jul 06 16:04:03.202698 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:03.202559 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" event={"ID":"97f409be-d908-4e0a-8488-c8f2a9703619","Type":"ContainerStarted","Data":"93ec7cd8dba49fc49e338ee64e3790f03a2cca8ae4a0ea4fbe8d0d7507aba4d1"} Jul 06 16:04:03.230596 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:03.230552 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" podStartSLOduration=1.230518875 podStartE2EDuration="1.230518875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:04:03.228826344 +0000 UTC m=+2264.861378326" watchObservedRunningTime="2026-07-06 16:04:03.230518875 +0000 UTC m=+2264.863070836" Jul 06 16:04:03.641405 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:03.641368 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:03.646917 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:03.646885 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:04.210690 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:04.210624 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:04.217263 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:04.217008 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:05.345795 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:05.345764 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:05.424877 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:05.424845 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:05.490264 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:05.490229 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:05.858637 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:05.858606 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:06.077361 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:06.077328 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:06.178135 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:06.178048 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:08.018664 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:08.018611 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:08.122839 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:08.122807 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:09.850258 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:09.850225 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:11.161245 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:11.161165 2561 patch_prober.go:28] interesting pod/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 06 16:04:11.161665 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:11.161239 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" podUID="c27deb48-2f82-405d-b6c4-13209d22b934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 06 16:04:11.268520 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:11.268488 2561 generic.go:356] "Generic (PLEG): container finished" podID="c27deb48-2f82-405d-b6c4-13209d22b934" containerID="e789b103a11680bd2b79f6c6a68821a906ebdb947ee818b371465112b99766cd" exitCode=0 Jul 06 16:04:11.268702 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:11.268576 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" event={"ID":"c27deb48-2f82-405d-b6c4-13209d22b934","Type":"ContainerDied","Data":"e789b103a11680bd2b79f6c6a68821a906ebdb947ee818b371465112b99766cd"} Jul 06 16:04:11.269041 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:11.269025 2561 scope.go:117] "RemoveContainer" containerID="e789b103a11680bd2b79f6c6a68821a906ebdb947ee818b371465112b99766cd" Jul 06 16:04:11.270036 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:11.270008 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:04:11.270092 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:11.270074 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:04:11.270136 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:11.270124 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:04:12.275454 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:12.275420 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" event={"ID":"c27deb48-2f82-405d-b6c4-13209d22b934","Type":"ContainerStarted","Data":"d2528f948bef50ce38aaa631cf949f898aa490048715a783a34e1b0d8bd80ed9"} Jul 06 16:04:12.462510 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:12.462469 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:04:12.462722 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:12.462622 2561 patch_prober.go:28] interesting pod/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 06 16:04:12.462722 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:12.462674 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" podUID="c27deb48-2f82-405d-b6c4-13209d22b934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 06 16:04:13.462512 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:13.462465 2561 patch_prober.go:28] interesting pod/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 06 16:04:13.462955 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:13.462556 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" podUID="c27deb48-2f82-405d-b6c4-13209d22b934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 06 16:04:14.462143 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:14.462098 2561 patch_prober.go:28] interesting pod/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 06 16:04:14.462332 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:14.462186 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" podUID="c27deb48-2f82-405d-b6c4-13209d22b934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 06 16:04:15.462076 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:15.462045 2561 patch_prober.go:28] interesting pod/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 06 16:04:15.462448 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:15.462099 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" podUID="c27deb48-2f82-405d-b6c4-13209d22b934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 06 16:04:15.741251 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:15.741218 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:16.467233 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:16.467203 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:04:16.467736 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:16.467711 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:04:16.468400 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:16.468381 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:04:18.799840 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:18.799813 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:18.900631 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:18.900600 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:19.211679 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:19.211647 2561 patch_prober.go:28] interesting pod/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 06 16:04:19.211876 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:19.211703 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" podUID="97f409be-d908-4e0a-8488-c8f2a9703619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 06 16:04:19.318787 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:19.318757 2561 generic.go:356] "Generic (PLEG): container finished" podID="97f409be-d908-4e0a-8488-c8f2a9703619" containerID="657f5a0a721a441acc3e5efdc7e0f9ec54ed978a97afd05ce6d7177cf49e3b73" exitCode=0 Jul 06 16:04:19.318949 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:19.318822 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" event={"ID":"97f409be-d908-4e0a-8488-c8f2a9703619","Type":"ContainerDied","Data":"657f5a0a721a441acc3e5efdc7e0f9ec54ed978a97afd05ce6d7177cf49e3b73"} Jul 06 16:04:19.319274 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:19.319255 2561 scope.go:117] "RemoveContainer" containerID="657f5a0a721a441acc3e5efdc7e0f9ec54ed978a97afd05ce6d7177cf49e3b73" Jul 06 16:04:19.320288 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:19.320252 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:04:19.320394 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:19.320332 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:04:19.320394 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:19.320373 2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892172Ki","pods":"250"} Jul 06 16:04:20.330939 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:20.330867 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" event={"ID":"97f409be-d908-4e0a-8488-c8f2a9703619","Type":"ContainerStarted","Data":"a7e4dd29ed27045e98a1a630d10e30b1cdb8d290b2f04a2a7e06a0e9efab2ab1"} Jul 06 16:04:20.641592 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:20.641460 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:20.641760 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:20.641688 2561 patch_prober.go:28] interesting pod/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 06 16:04:20.641760 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:20.641744 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" podUID="97f409be-d908-4e0a-8488-c8f2a9703619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 06 16:04:21.641335 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:21.641301 2561 patch_prober.go:28] interesting pod/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 06 16:04:21.641741 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:21.641364 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" podUID="97f409be-d908-4e0a-8488-c8f2a9703619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 06 16:04:22.641370 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:22.641333 2561 patch_prober.go:28] interesting pod/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 06 16:04:22.641753 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:22.641404 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" podUID="97f409be-d908-4e0a-8488-c8f2a9703619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 06 16:04:23.641447 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:23.641414 2561 patch_prober.go:28] interesting pod/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 06 16:04:23.641909 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:23.641474 2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" podUID="97f409be-d908-4e0a-8488-c8f2a9703619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 06 16:04:24.645152 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:24.645121 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:24.645616 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:24.645461 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:24.646197 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:24.646179 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:04:26.872108 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:26.872078 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:27.001784 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:27.001754 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:45.379663 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:04:45.379628 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:05:32.739968 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:05:32.739938 2561 generic.go:356] "Generic (PLEG): container finished" podID="46d03c46-7d6d-4084-8b9a-211fec671746" containerID="6a6df0629fbff2b5d924312bf6dad4258b504ae6cdb0e4a923147166adaf0281" exitCode=1 Jul 06 16:05:32.740358 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:05:32.740016 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" event={"ID":"46d03c46-7d6d-4084-8b9a-211fec671746","Type":"ContainerDied","Data":"6a6df0629fbff2b5d924312bf6dad4258b504ae6cdb0e4a923147166adaf0281"} Jul 06 16:05:32.740358 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:05:32.740342 2561 scope.go:117] "RemoveContainer" containerID="6a6df0629fbff2b5d924312bf6dad4258b504ae6cdb0e4a923147166adaf0281" Jul 06 16:05:33.747624 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:05:33.747592 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" event={"ID":"46d03c46-7d6d-4084-8b9a-211fec671746","Type":"ContainerStarted","Data":"351b894bf0900457f66062fc3edc83cd6d37fd0e8db517b3a3baff7ce9e6481b"} Jul 06 16:05:33.747997 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:05:33.747795 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" Jul 06 16:05:46.457246 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:05:46.457214 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:05:48.361189 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:05:48.361154 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:00.192866 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:06:00.192829 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:04.755878 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:06:04.755852 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vhd2b" Jul 06 16:06:08.133000 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:06:08.132958 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:08.227718 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:06:08.227689 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:09.326158 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:06:09.326126 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:09.394857 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:06:09.394825 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:18.945508 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:06:18.945485 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 16:06:18.955425 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:06:18.955400 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 16:06:31.495183 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:06:31.495140 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:31.561355 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:06:31.561327 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:39.684012 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:06:39.683882 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:40.580926 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:06:40.580883 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:44.862330 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:06:44.862299 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:51.201102 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:06:51.201074 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:51.337795 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:06:51.337763 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:19.925652 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:07:19.925619 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:08:05.103610 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:08:05.103572 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:08:06.046377 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:08:06.046338 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:08:13.938890 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:08:13.938808 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:08:43.402413 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:08:43.402382 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:16.878450 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:09:16.878410 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:19.719327 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:09:19.719295 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:19.824521 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:09:19.824485 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:22.566364 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:09:22.566328 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:27.735180 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:09:27.735137 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:27.826875 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:09:27.826836 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:40.109780 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:09:40.109692 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:41.021965 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:09:41.021913 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:42.115265 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:09:42.115233 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:42.585732 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:09:42.585698 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:42.824117 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:09:42.824082 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:42.952846 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:09:42.952764 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:43.245335 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:09:43.245265 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:47.736806 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:09:47.736767 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:27.118209 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:27.118175 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:27.449012 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:27.448934 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6"] Jul 06 16:10:27.449196 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:27.449154 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" podUID="c27deb48-2f82-405d-b6c4-13209d22b934" containerName="istio-proxy" containerID="cri-o://d2528f948bef50ce38aaa631cf949f898aa490048715a783a34e1b0d8bd80ed9" gracePeriod=30 Jul 06 16:10:27.505826 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:27.505796 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:28.455835 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:28.455802 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:28.581298 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:28.581268 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:28.673256 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:28.673230 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:28.905100 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:28.905069 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:29.007326 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:29.007291 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:29.944846 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:29.944816 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:31.470561 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:31.470514 2561 patch_prober.go:28] interesting pod/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 16:10:31.470918 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:31.470596 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" podUID="c27deb48-2f82-405d-b6c4-13209d22b934" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 16:10:32.706288 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.706235 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:10:32.833807 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.833780 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-istio-data\") pod \"c27deb48-2f82-405d-b6c4-13209d22b934\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " Jul 06 16:10:32.833975 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.833858 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27deb48-2f82-405d-b6c4-13209d22b934-istio-podinfo\") pod \"c27deb48-2f82-405d-b6c4-13209d22b934\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " Jul 06 16:10:32.833975 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.833889 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zb2cv\" (UniqueName: \"kubernetes.io/projected/c27deb48-2f82-405d-b6c4-13209d22b934-kube-api-access-zb2cv\") pod \"c27deb48-2f82-405d-b6c4-13209d22b934\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " Jul 06 16:10:32.833975 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.833907 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27deb48-2f82-405d-b6c4-13209d22b934-istiod-ca-cert\") pod \"c27deb48-2f82-405d-b6c4-13209d22b934\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " Jul 06 16:10:32.833975 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.833926 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-workload-socket\") pod \"c27deb48-2f82-405d-b6c4-13209d22b934\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " Jul 06 16:10:32.834204 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.833976 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27deb48-2f82-405d-b6c4-13209d22b934-istio-token\") pod \"c27deb48-2f82-405d-b6c4-13209d22b934\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " Jul 06 16:10:32.834204 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.834012 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-credential-socket\") pod \"c27deb48-2f82-405d-b6c4-13209d22b934\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " Jul 06 16:10:32.834204 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.834067 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-workload-certs\") pod \"c27deb48-2f82-405d-b6c4-13209d22b934\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " Jul 06 16:10:32.834204 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.834138 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-istio-envoy\") pod \"c27deb48-2f82-405d-b6c4-13209d22b934\" (UID: \"c27deb48-2f82-405d-b6c4-13209d22b934\") " Jul 06 16:10:32.834399 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.834231 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c27deb48-2f82-405d-b6c4-13209d22b934" (UID: "c27deb48-2f82-405d-b6c4-13209d22b934"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:10:32.834399 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.834283 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27deb48-2f82-405d-b6c4-13209d22b934-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c27deb48-2f82-405d-b6c4-13209d22b934" (UID: "c27deb48-2f82-405d-b6c4-13209d22b934"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:10:32.834522 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.834499 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c27deb48-2f82-405d-b6c4-13209d22b934" (UID: "c27deb48-2f82-405d-b6c4-13209d22b934"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:10:32.834750 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.834730 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27deb48-2f82-405d-b6c4-13209d22b934-istiod-ca-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:32.834882 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.834791 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-workload-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:32.834882 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.834807 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-credential-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:32.834882 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.834739 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c27deb48-2f82-405d-b6c4-13209d22b934" (UID: "c27deb48-2f82-405d-b6c4-13209d22b934"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:10:32.834882 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.834818 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-istio-data" (OuterVolumeSpecName: "istio-data") pod "c27deb48-2f82-405d-b6c4-13209d22b934" (UID: "c27deb48-2f82-405d-b6c4-13209d22b934"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:10:32.836249 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.836226 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c27deb48-2f82-405d-b6c4-13209d22b934-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c27deb48-2f82-405d-b6c4-13209d22b934" (UID: "c27deb48-2f82-405d-b6c4-13209d22b934"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:10:32.836413 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.836398 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27deb48-2f82-405d-b6c4-13209d22b934-istio-token" (OuterVolumeSpecName: "istio-token") pod "c27deb48-2f82-405d-b6c4-13209d22b934" (UID: "c27deb48-2f82-405d-b6c4-13209d22b934"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:10:32.836839 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.836821 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c27deb48-2f82-405d-b6c4-13209d22b934" (UID: "c27deb48-2f82-405d-b6c4-13209d22b934"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:10:32.836910 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.836843 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27deb48-2f82-405d-b6c4-13209d22b934-kube-api-access-zb2cv" (OuterVolumeSpecName: "kube-api-access-zb2cv") pod "c27deb48-2f82-405d-b6c4-13209d22b934" (UID: "c27deb48-2f82-405d-b6c4-13209d22b934"). InnerVolumeSpecName "kube-api-access-zb2cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:10:32.936315 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.936291 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-workload-certs\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:32.936315 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.936317 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-istio-envoy\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:32.936459 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.936325 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27deb48-2f82-405d-b6c4-13209d22b934-istio-data\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:32.936459 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.936333 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27deb48-2f82-405d-b6c4-13209d22b934-istio-podinfo\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:32.936459 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.936341 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb2cv\" (UniqueName: \"kubernetes.io/projected/c27deb48-2f82-405d-b6c4-13209d22b934-kube-api-access-zb2cv\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:32.936459 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:32.936351 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27deb48-2f82-405d-b6c4-13209d22b934-istio-token\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:33.315173 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:33.315143 2561 generic.go:356] "Generic (PLEG): container finished" podID="c27deb48-2f82-405d-b6c4-13209d22b934" containerID="d2528f948bef50ce38aaa631cf949f898aa490048715a783a34e1b0d8bd80ed9" exitCode=0 Jul 06 16:10:33.315379 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:33.315200 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" Jul 06 16:10:33.315379 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:33.315235 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" event={"ID":"c27deb48-2f82-405d-b6c4-13209d22b934","Type":"ContainerDied","Data":"d2528f948bef50ce38aaa631cf949f898aa490048715a783a34e1b0d8bd80ed9"} Jul 06 16:10:33.315379 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:33.315301 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6" event={"ID":"c27deb48-2f82-405d-b6c4-13209d22b934","Type":"ContainerDied","Data":"e6c09559cecbab511fab68bbb8e8e60ca5c07d75c546224e2c09067da384ae9f"} Jul 06 16:10:33.315379 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:33.315334 2561 scope.go:117] "RemoveContainer" containerID="d2528f948bef50ce38aaa631cf949f898aa490048715a783a34e1b0d8bd80ed9" Jul 06 16:10:33.325931 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:33.325905 2561 scope.go:117] "RemoveContainer" containerID="e789b103a11680bd2b79f6c6a68821a906ebdb947ee818b371465112b99766cd" Jul 06 16:10:33.326978 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:33.326901 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:33.335319 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:33.335302 2561 scope.go:117] "RemoveContainer" containerID="d2528f948bef50ce38aaa631cf949f898aa490048715a783a34e1b0d8bd80ed9" Jul 06 16:10:33.335576 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:10:33.335559 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2528f948bef50ce38aaa631cf949f898aa490048715a783a34e1b0d8bd80ed9\": container with ID starting with d2528f948bef50ce38aaa631cf949f898aa490048715a783a34e1b0d8bd80ed9 not found: ID does not exist" containerID="d2528f948bef50ce38aaa631cf949f898aa490048715a783a34e1b0d8bd80ed9" Jul 06 16:10:33.335643 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:33.335584 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2528f948bef50ce38aaa631cf949f898aa490048715a783a34e1b0d8bd80ed9"} err="failed to get container status \"d2528f948bef50ce38aaa631cf949f898aa490048715a783a34e1b0d8bd80ed9\": rpc error: code = NotFound desc = could not find container \"d2528f948bef50ce38aaa631cf949f898aa490048715a783a34e1b0d8bd80ed9\": container with ID starting with d2528f948bef50ce38aaa631cf949f898aa490048715a783a34e1b0d8bd80ed9 not found: ID does not exist" Jul 06 16:10:33.335643 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:33.335601 2561 scope.go:117] "RemoveContainer" containerID="e789b103a11680bd2b79f6c6a68821a906ebdb947ee818b371465112b99766cd" Jul 06 16:10:33.335790 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:10:33.335775 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e789b103a11680bd2b79f6c6a68821a906ebdb947ee818b371465112b99766cd\": container with ID starting with e789b103a11680bd2b79f6c6a68821a906ebdb947ee818b371465112b99766cd not found: ID does not exist" containerID="e789b103a11680bd2b79f6c6a68821a906ebdb947ee818b371465112b99766cd" Jul 06 16:10:33.335827 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:33.335797 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e789b103a11680bd2b79f6c6a68821a906ebdb947ee818b371465112b99766cd"} err="failed to get container status \"e789b103a11680bd2b79f6c6a68821a906ebdb947ee818b371465112b99766cd\": rpc error: code = NotFound desc = could not find container \"e789b103a11680bd2b79f6c6a68821a906ebdb947ee818b371465112b99766cd\": container with ID starting with e789b103a11680bd2b79f6c6a68821a906ebdb947ee818b371465112b99766cd not found: ID does not exist" Jul 06 16:10:33.350971 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:33.350619 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6"] Jul 06 16:10:33.354832 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:33.354812 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-b75322-openshift-default-86cb8564ff-xvxj6"] Jul 06 16:10:33.412072 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:33.412047 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:34.801723 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:34.801691 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c27deb48-2f82-405d-b6c4-13209d22b934" path="/var/lib/kubelet/pods/c27deb48-2f82-405d-b6c4-13209d22b934/volumes" Jul 06 16:10:36.447599 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:36.447565 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:36.545593 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:36.545557 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:36.723254 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:36.723217 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l"] Jul 06 16:10:36.723477 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:36.723428 2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" podUID="97f409be-d908-4e0a-8488-c8f2a9703619" containerName="istio-proxy" containerID="cri-o://a7e4dd29ed27045e98a1a630d10e30b1cdb8d290b2f04a2a7e06a0e9efab2ab1" gracePeriod=30 Jul 06 16:10:36.749147 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:36.749114 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:36.975405 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:36.975316 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:39.647680 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:39.647648 2561 patch_prober.go:28] interesting pod/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 16:10:39.648067 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:39.647698 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" podUID="97f409be-d908-4e0a-8488-c8f2a9703619" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 16:10:41.981105 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:41.981080 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:10:42.018711 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.018646 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-istio-data\") pod \"97f409be-d908-4e0a-8488-c8f2a9703619\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " Jul 06 16:10:42.018711 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.018686 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-istio-envoy\") pod \"97f409be-d908-4e0a-8488-c8f2a9703619\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " Jul 06 16:10:42.018711 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.018708 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f409be-d908-4e0a-8488-c8f2a9703619-istio-podinfo\") pod \"97f409be-d908-4e0a-8488-c8f2a9703619\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " Jul 06 16:10:42.018934 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.018744 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-workload-socket\") pod \"97f409be-d908-4e0a-8488-c8f2a9703619\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " Jul 06 16:10:42.018934 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.018761 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-workload-certs\") pod \"97f409be-d908-4e0a-8488-c8f2a9703619\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " Jul 06 16:10:42.018934 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.018778 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f409be-d908-4e0a-8488-c8f2a9703619-istiod-ca-cert\") pod \"97f409be-d908-4e0a-8488-c8f2a9703619\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " Jul 06 16:10:42.018934 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.018822 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9dzk\" (UniqueName: \"kubernetes.io/projected/97f409be-d908-4e0a-8488-c8f2a9703619-kube-api-access-f9dzk\") pod \"97f409be-d908-4e0a-8488-c8f2a9703619\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " Jul 06 16:10:42.019191 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.019080 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "97f409be-d908-4e0a-8488-c8f2a9703619" (UID: "97f409be-d908-4e0a-8488-c8f2a9703619"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:10:42.019191 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.019101 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "97f409be-d908-4e0a-8488-c8f2a9703619" (UID: "97f409be-d908-4e0a-8488-c8f2a9703619"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:10:42.019191 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.019149 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-credential-socket\") pod \"97f409be-d908-4e0a-8488-c8f2a9703619\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " Jul 06 16:10:42.019346 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.019269 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f409be-d908-4e0a-8488-c8f2a9703619-istio-token\") pod \"97f409be-d908-4e0a-8488-c8f2a9703619\" (UID: \"97f409be-d908-4e0a-8488-c8f2a9703619\") " Jul 06 16:10:42.019787 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.019397 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "97f409be-d908-4e0a-8488-c8f2a9703619" (UID: "97f409be-d908-4e0a-8488-c8f2a9703619"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:10:42.019787 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.019579 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f409be-d908-4e0a-8488-c8f2a9703619-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "97f409be-d908-4e0a-8488-c8f2a9703619" (UID: "97f409be-d908-4e0a-8488-c8f2a9703619"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:10:42.019787 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.019645 2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-workload-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:42.019787 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.019663 2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-workload-certs\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:42.019787 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.019677 2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f409be-d908-4e0a-8488-c8f2a9703619-istiod-ca-cert\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:42.019787 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.019689 2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-credential-socket\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:42.019787 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.019742 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-istio-data" (OuterVolumeSpecName: "istio-data") pod "97f409be-d908-4e0a-8488-c8f2a9703619" (UID: "97f409be-d908-4e0a-8488-c8f2a9703619"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:10:42.021173 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.021152 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "97f409be-d908-4e0a-8488-c8f2a9703619" (UID: "97f409be-d908-4e0a-8488-c8f2a9703619"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:10:42.021173 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.021152 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f409be-d908-4e0a-8488-c8f2a9703619-kube-api-access-f9dzk" (OuterVolumeSpecName: "kube-api-access-f9dzk") pod "97f409be-d908-4e0a-8488-c8f2a9703619" (UID: "97f409be-d908-4e0a-8488-c8f2a9703619"). InnerVolumeSpecName "kube-api-access-f9dzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:10:42.021341 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.021210 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97f409be-d908-4e0a-8488-c8f2a9703619-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "97f409be-d908-4e0a-8488-c8f2a9703619" (UID: "97f409be-d908-4e0a-8488-c8f2a9703619"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:10:42.021617 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.021594 2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f409be-d908-4e0a-8488-c8f2a9703619-istio-token" (OuterVolumeSpecName: "istio-token") pod "97f409be-d908-4e0a-8488-c8f2a9703619" (UID: "97f409be-d908-4e0a-8488-c8f2a9703619"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:10:42.120911 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.120877 2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-istio-data\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:42.120911 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.120905 2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f409be-d908-4e0a-8488-c8f2a9703619-istio-envoy\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:42.120911 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.120913 2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f409be-d908-4e0a-8488-c8f2a9703619-istio-podinfo\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:42.121119 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.120921 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9dzk\" (UniqueName: \"kubernetes.io/projected/97f409be-d908-4e0a-8488-c8f2a9703619-kube-api-access-f9dzk\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:42.121119 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.120930 2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f409be-d908-4e0a-8488-c8f2a9703619-istio-token\") on node \"ip-10-0-129-225.ec2.internal\" DevicePath \"\"" Jul 06 16:10:42.366936 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.366855 2561 generic.go:356] "Generic (PLEG): container finished" podID="97f409be-d908-4e0a-8488-c8f2a9703619" containerID="a7e4dd29ed27045e98a1a630d10e30b1cdb8d290b2f04a2a7e06a0e9efab2ab1" exitCode=0 Jul 06 16:10:42.366936 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.366902 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" Jul 06 16:10:42.367112 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.366944 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" event={"ID":"97f409be-d908-4e0a-8488-c8f2a9703619","Type":"ContainerDied","Data":"a7e4dd29ed27045e98a1a630d10e30b1cdb8d290b2f04a2a7e06a0e9efab2ab1"} Jul 06 16:10:42.367112 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.366997 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l" event={"ID":"97f409be-d908-4e0a-8488-c8f2a9703619","Type":"ContainerDied","Data":"93ec7cd8dba49fc49e338ee64e3790f03a2cca8ae4a0ea4fbe8d0d7507aba4d1"} Jul 06 16:10:42.367112 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.367026 2561 scope.go:117] "RemoveContainer" containerID="a7e4dd29ed27045e98a1a630d10e30b1cdb8d290b2f04a2a7e06a0e9efab2ab1" Jul 06 16:10:42.378399 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.378378 2561 scope.go:117] "RemoveContainer" containerID="657f5a0a721a441acc3e5efdc7e0f9ec54ed978a97afd05ce6d7177cf49e3b73" Jul 06 16:10:42.387408 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.387389 2561 scope.go:117] "RemoveContainer" containerID="a7e4dd29ed27045e98a1a630d10e30b1cdb8d290b2f04a2a7e06a0e9efab2ab1" Jul 06 16:10:42.387694 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:10:42.387675 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e4dd29ed27045e98a1a630d10e30b1cdb8d290b2f04a2a7e06a0e9efab2ab1\": container with ID starting with a7e4dd29ed27045e98a1a630d10e30b1cdb8d290b2f04a2a7e06a0e9efab2ab1 not found: ID does not exist" containerID="a7e4dd29ed27045e98a1a630d10e30b1cdb8d290b2f04a2a7e06a0e9efab2ab1" Jul 06 16:10:42.387740 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.387703 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e4dd29ed27045e98a1a630d10e30b1cdb8d290b2f04a2a7e06a0e9efab2ab1"} err="failed to get container status \"a7e4dd29ed27045e98a1a630d10e30b1cdb8d290b2f04a2a7e06a0e9efab2ab1\": rpc error: code = NotFound desc = could not find container \"a7e4dd29ed27045e98a1a630d10e30b1cdb8d290b2f04a2a7e06a0e9efab2ab1\": container with ID starting with a7e4dd29ed27045e98a1a630d10e30b1cdb8d290b2f04a2a7e06a0e9efab2ab1 not found: ID does not exist" Jul 06 16:10:42.387740 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.387721 2561 scope.go:117] "RemoveContainer" containerID="657f5a0a721a441acc3e5efdc7e0f9ec54ed978a97afd05ce6d7177cf49e3b73" Jul 06 16:10:42.387997 ip-10-0-129-225 kubenswrapper[2561]: E0706 16:10:42.387980 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657f5a0a721a441acc3e5efdc7e0f9ec54ed978a97afd05ce6d7177cf49e3b73\": container with ID starting with 657f5a0a721a441acc3e5efdc7e0f9ec54ed978a97afd05ce6d7177cf49e3b73 not found: ID does not exist" containerID="657f5a0a721a441acc3e5efdc7e0f9ec54ed978a97afd05ce6d7177cf49e3b73" Jul 06 16:10:42.388043 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.388003 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657f5a0a721a441acc3e5efdc7e0f9ec54ed978a97afd05ce6d7177cf49e3b73"} err="failed to get container status \"657f5a0a721a441acc3e5efdc7e0f9ec54ed978a97afd05ce6d7177cf49e3b73\": rpc error: code = NotFound desc = could not find container \"657f5a0a721a441acc3e5efdc7e0f9ec54ed978a97afd05ce6d7177cf49e3b73\": container with ID starting with 657f5a0a721a441acc3e5efdc7e0f9ec54ed978a97afd05ce6d7177cf49e3b73 not found: ID does not exist" Jul 06 16:10:42.404519 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.404452 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l"] Jul 06 16:10:42.418549 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.418505 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-690f40-openshift-default-847b4dd8c9-cgs2l"] Jul 06 16:10:42.490941 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.490921 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-nktks_0bb58fc6-57b0-4511-bb64-ca0a43d05ee0/manager/0.log" Jul 06 16:10:42.619220 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.619151 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vhd2b_46d03c46-7d6d-4084-8b9a-211fec671746/manager/1.log" Jul 06 16:10:42.774169 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.774145 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-68d58c4db6-p9kws_3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0/maas-api/0.log" Jul 06 16:10:42.799804 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.799775 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97f409be-d908-4e0a-8488-c8f2a9703619" path="/var/lib/kubelet/pods/97f409be-d908-4e0a-8488-c8f2a9703619/volumes" Jul 06 16:10:42.902726 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:42.902658 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722545-qqmh5_97ef6680-dccc-42fa-9521-346b03544ab0/cleanup/0.log" Jul 06 16:10:43.042137 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:43.042113 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722560-skpg7_8d951172-87e6-472e-a117-9210a1a7f20a/cleanup/0.log" Jul 06 16:10:43.170585 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:43.170483 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-957b6d485-2jkhc_ecf4090d-7bf0-4bff-a45b-9f91518ce855/manager/0.log" Jul 06 16:10:43.958401 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:43.958375 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bdc85857f-f2l8c_36396d1c-372d-4146-94a7-59e1c7003e40/manager/0.log" Jul 06 16:10:43.982761 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:43.982733 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bdc85857f-f2l8c_36396d1c-372d-4146-94a7-59e1c7003e40/copy-manifests/0.log" Jul 06 16:10:44.144595 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:44.144571 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-kmhdx_19f75d9b-f72a-4dea-ab21-87a689803e9b/postgres/0.log" Jul 06 16:10:45.289588 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:45.289557 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:45.391306 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:45.391275 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:45.864074 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:45.864043 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-z4kkt_3b1bc0d4-54a1-4658-9597-1451b967cdea/manager/0.log" Jul 06 16:10:46.106083 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:46.106050 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-vnlc6_c97007f0-45d4-42a8-b9b7-cab3728abaf0/registry-server/0.log" Jul 06 16:10:46.233547 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:46.233503 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-tgcnq_5bcbde0f-bbc4-4c5f-a089-daaeb038744d/manager/0.log" Jul 06 16:10:46.855936 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:46.855906 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4gpw22_874c3000-ae24-4343-9385-7e7e79ccfa33/istio-proxy/0.log" Jul 06 16:10:47.328869 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:47.328841 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-5nsgf_d9acfb2a-5f27-4ead-8402-a7a9d253d5ed/istio-proxy/1.log" Jul 06 16:10:47.578925 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:47.578855 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lr7lt_82d1b0f6-6ce2-46bf-acc3-61d410363776/payload-processing/0.log" Jul 06 16:10:48.155577 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:48.155552 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-jqhkh_fe9f4afd-8eff-411d-a9cb-8c93ee3ec521/main/0.log" Jul 06 16:10:53.164737 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:53.164693 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:55.256231 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:55.256179 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:57.113932 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:57.113898 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-77mbr_b4c84634-a9f3-4cc5-a67b-09805e8f67af/global-pull-secret-syncer/0.log" Jul 06 16:10:57.292209 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:57.292163 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" Jul 06 16:10:57.432879 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:10:57.432803 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-225.ec2.internal_91375dee49182778f247ddc7d3913f13/haproxy/0.log" Jul 06 16:11:02.785620 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:02.785584 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-z4kkt_3b1bc0d4-54a1-4658-9597-1451b967cdea/manager/0.log" Jul 06 16:11:02.879185 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:02.879105 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-vnlc6_c97007f0-45d4-42a8-b9b7-cab3728abaf0/registry-server/0.log" Jul 06 16:11:03.070929 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:03.070895 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-tgcnq_5bcbde0f-bbc4-4c5f-a089-daaeb038744d/manager/0.log" Jul 06 16:11:05.336007 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:05.335975 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-np2l9_5369321f-60a5-4cd6-bdad-9ae4e17f436d/monitoring-plugin/0.log" Jul 06 16:11:05.388345 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:05.388322 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5wrkh_c3474713-76cd-441b-97a4-d7244637c9fb/node-exporter/0.log" Jul 06 16:11:05.436326 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:05.436307 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5wrkh_c3474713-76cd-441b-97a4-d7244637c9fb/kube-rbac-proxy/0.log" Jul 06 16:11:05.495384 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:05.495364 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5wrkh_c3474713-76cd-441b-97a4-d7244637c9fb/init-textfile/0.log" Jul 06 16:11:05.983110 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:05.983081 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0f8157a4-8adc-4a02-a4ca-c4b2989a7a30/prometheus/0.log" Jul 06 16:11:06.060891 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:06.060867 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0f8157a4-8adc-4a02-a4ca-c4b2989a7a30/config-reloader/0.log" Jul 06 16:11:06.095457 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:06.095423 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0f8157a4-8adc-4a02-a4ca-c4b2989a7a30/thanos-sidecar/0.log" Jul 06 16:11:06.137867 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:06.137841 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0f8157a4-8adc-4a02-a4ca-c4b2989a7a30/kube-rbac-proxy-web/0.log" Jul 06 16:11:06.188121 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:06.188095 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0f8157a4-8adc-4a02-a4ca-c4b2989a7a30/kube-rbac-proxy/0.log" Jul 06 16:11:06.241162 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:06.241092 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0f8157a4-8adc-4a02-a4ca-c4b2989a7a30/kube-rbac-proxy-thanos/0.log" Jul 06 16:11:06.281224 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:06.281204 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0f8157a4-8adc-4a02-a4ca-c4b2989a7a30/init-config-reloader/0.log" Jul 06 16:11:06.326027 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:06.326010 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-r6mpw_cb408066-218e-4efb-8407-ccf6c8df3994/prometheus-operator/0.log" Jul 06 16:11:06.354342 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:06.354322 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-r6mpw_cb408066-218e-4efb-8407-ccf6c8df3994/kube-rbac-proxy/0.log" Jul 06 16:11:06.416955 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:06.416911 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-696bd4df5b-9xbj7_da5e8b26-1ec1-419e-af70-3d9947dd2c52/telemeter-client/0.log" Jul 06 16:11:06.456907 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:06.456886 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-696bd4df5b-9xbj7_da5e8b26-1ec1-419e-af70-3d9947dd2c52/reload/0.log" Jul 06 16:11:06.483119 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:06.483096 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-696bd4df5b-9xbj7_da5e8b26-1ec1-419e-af70-3d9947dd2c52/kube-rbac-proxy/0.log" Jul 06 16:11:07.414780 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:07.414748 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lhbjw_6fa66968-290f-4570-b983-15d1d692f9b5/konnectivity-agent/0.log" limit=1024 Jul 06 16:11:08.002963 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.002933 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj"] Jul 06 16:11:08.003335 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.003322 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97f409be-d908-4e0a-8488-c8f2a9703619" containerName="istio-proxy" Jul 06 16:11:08.003387 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.003336 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="97f409be-d908-4e0a-8488-c8f2a9703619" containerName="istio-proxy" Jul 06 16:11:08.003387 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.003352 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97f409be-d908-4e0a-8488-c8f2a9703619" containerName="istio-proxy" Jul 06 16:11:08.003387 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.003358 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="97f409be-d908-4e0a-8488-c8f2a9703619" containerName="istio-proxy" Jul 06 16:11:08.003387 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.003365 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27deb48-2f82-405d-b6c4-13209d22b934" containerName="istio-proxy" Jul 06 16:11:08.003387 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.003370 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27deb48-2f82-405d-b6c4-13209d22b934" containerName="istio-proxy" Jul 06 16:11:08.003387 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.003376 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27deb48-2f82-405d-b6c4-13209d22b934" containerName="istio-proxy" Jul 06 16:11:08.003387 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.003381 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27deb48-2f82-405d-b6c4-13209d22b934" containerName="istio-proxy" Jul 06 16:11:08.003651 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.003443 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27deb48-2f82-405d-b6c4-13209d22b934" containerName="istio-proxy" Jul 06 16:11:08.003651 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.003453 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="97f409be-d908-4e0a-8488-c8f2a9703619" containerName="istio-proxy" Jul 06 16:11:08.006805 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.006782 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.009923 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.009901 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-96f6w\"/\"default-dockercfg-msmgz\"" Jul 06 16:11:08.010036 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.009902 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-96f6w\"/\"openshift-service-ca.crt\"" Jul 06 16:11:08.011253 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.011234 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-96f6w\"/\"kube-root-ca.crt\"" Jul 06 16:11:08.017113 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.017093 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj"] Jul 06 16:11:08.146066 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.146042 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fdac8e9e-e8b5-4694-aae6-dc3ab57b010b-podres\") pod \"perf-node-gather-daemonset-2ssbj\" (UID: \"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b\") " pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.146182 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.146108 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2zpn\" (UniqueName: \"kubernetes.io/projected/fdac8e9e-e8b5-4694-aae6-dc3ab57b010b-kube-api-access-n2zpn\") pod \"perf-node-gather-daemonset-2ssbj\" (UID: \"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b\") " pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.146182 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.146127 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdac8e9e-e8b5-4694-aae6-dc3ab57b010b-sys\") pod \"perf-node-gather-daemonset-2ssbj\" (UID: \"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b\") " pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.146182 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.146151 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdac8e9e-e8b5-4694-aae6-dc3ab57b010b-lib-modules\") pod \"perf-node-gather-daemonset-2ssbj\" (UID: \"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b\") " pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.146288 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.146188 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fdac8e9e-e8b5-4694-aae6-dc3ab57b010b-proc\") pod \"perf-node-gather-daemonset-2ssbj\" (UID: \"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b\") " pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.247626 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.247593 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fdac8e9e-e8b5-4694-aae6-dc3ab57b010b-podres\") pod \"perf-node-gather-daemonset-2ssbj\" (UID: \"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b\") " pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.247737 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.247667 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2zpn\" (UniqueName: \"kubernetes.io/projected/fdac8e9e-e8b5-4694-aae6-dc3ab57b010b-kube-api-access-n2zpn\") pod \"perf-node-gather-daemonset-2ssbj\" (UID: \"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b\") " pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.247737 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.247694 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdac8e9e-e8b5-4694-aae6-dc3ab57b010b-sys\") pod \"perf-node-gather-daemonset-2ssbj\" (UID: \"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b\") " pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.247737 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.247728 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdac8e9e-e8b5-4694-aae6-dc3ab57b010b-lib-modules\") pod \"perf-node-gather-daemonset-2ssbj\" (UID: \"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b\") " pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.247850 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.247772 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fdac8e9e-e8b5-4694-aae6-dc3ab57b010b-podres\") pod \"perf-node-gather-daemonset-2ssbj\" (UID: \"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b\") " pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.247850 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.247795 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fdac8e9e-e8b5-4694-aae6-dc3ab57b010b-proc\") pod \"perf-node-gather-daemonset-2ssbj\" (UID: \"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b\") " pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.247850 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.247800 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdac8e9e-e8b5-4694-aae6-dc3ab57b010b-sys\") pod \"perf-node-gather-daemonset-2ssbj\" (UID: \"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b\") " pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.247945 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.247865 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdac8e9e-e8b5-4694-aae6-dc3ab57b010b-lib-modules\") pod \"perf-node-gather-daemonset-2ssbj\" (UID: \"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b\") " pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.247945 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.247875 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fdac8e9e-e8b5-4694-aae6-dc3ab57b010b-proc\") pod \"perf-node-gather-daemonset-2ssbj\" (UID: \"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b\") " pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.257062 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.257011 2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2zpn\" (UniqueName: \"kubernetes.io/projected/fdac8e9e-e8b5-4694-aae6-dc3ab57b010b-kube-api-access-n2zpn\") pod \"perf-node-gather-daemonset-2ssbj\" (UID: \"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b\") " pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.318317 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.318296 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:08.460198 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.460037 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj"] Jul 06 16:11:08.463059 ip-10-0-129-225 kubenswrapper[2561]: W0706 16:11:08.463032 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfdac8e9e_e8b5_4694_aae6_dc3ab57b010b.slice/crio-fd8211da67c38c6c6f92c8d114ee7bf8869a619b6c7f4c9cded5009a153cd7f0 WatchSource:0}: Error finding container fd8211da67c38c6c6f92c8d114ee7bf8869a619b6c7f4c9cded5009a153cd7f0: Status 404 returned error can't find the container with id fd8211da67c38c6c6f92c8d114ee7bf8869a619b6c7f4c9cded5009a153cd7f0 Jul 06 16:11:08.506115 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.506083 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" event={"ID":"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b","Type":"ContainerStarted","Data":"fd8211da67c38c6c6f92c8d114ee7bf8869a619b6c7f4c9cded5009a153cd7f0"} Jul 06 16:11:08.625808 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.625782 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 16:11:08.631651 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:08.631631 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/2.log" Jul 06 16:11:09.190141 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:09.190115 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-686ff6c96c-fkk4w_a9d48bff-9a51-48a7-b5f0-30f36e377202/console/0.log" Jul 06 16:11:09.235444 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:09.235414 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-2qq7v_55501abc-cc50-4838-b8ac-c28284b253f0/download-server/0.log" Jul 06 16:11:09.512496 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:09.512460 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" event={"ID":"fdac8e9e-e8b5-4694-aae6-dc3ab57b010b","Type":"ContainerStarted","Data":"669415fc8be4b67ec17dafdc34ada568f60f0c351548ab7478f24b0431a2b54e"} Jul 06 16:11:09.512934 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:09.512545 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:09.550802 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:09.550756 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" podStartSLOduration=2.550742007 podStartE2EDuration="2.550742007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:11:09.549969816 +0000 UTC m=+2691.182521798" watchObservedRunningTime="2026-07-06 16:11:09.550742007 +0000 UTC m=+2691.183293990" Jul 06 16:11:11.256034 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:11.256007 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6bhc8_cb50cb21-f13c-4d8c-ac8a-088425ccbd50/dns/0.log" Jul 06 16:11:11.313928 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:11.313888 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6bhc8_cb50cb21-f13c-4d8c-ac8a-088425ccbd50/kube-rbac-proxy/0.log" Jul 06 16:11:11.645874 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:11.645797 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w8l5z_75fcdf5f-2f1c-4a7c-8d40-07df5060fbb6/dns-node-resolver/0.log" Jul 06 16:11:12.359826 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:12.359798 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c6fd88d46-wfqsl_13d38e19-fbaf-4051-9062-64ab5d08eea6/registry/0.log" Jul 06 16:11:12.484065 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:12.484037 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w44v2_3b2617bf-d392-4486-ab8c-3d0aa5f481aa/node-ca/0.log" Jul 06 16:11:13.604136 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:13.604111 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4gpw22_874c3000-ae24-4343-9385-7e7e79ccfa33/istio-proxy/0.log" Jul 06 16:11:13.853127 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:13.853095 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-5nsgf_d9acfb2a-5f27-4ead-8402-a7a9d253d5ed/istio-proxy/0.log" Jul 06 16:11:13.877886 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:13.877812 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-5nsgf_d9acfb2a-5f27-4ead-8402-a7a9d253d5ed/istio-proxy/1.log" Jul 06 16:11:13.967920 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:13.967891 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lr7lt_82d1b0f6-6ce2-46bf-acc3-61d410363776/payload-processing/0.log" Jul 06 16:11:14.884402 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:14.884376 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lrpnb_fffc4cf2-1b41-40c5-943f-f01cc52933b7/serve-healthcheck-canary/0.log" Jul 06 16:11:15.444773 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:15.444740 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-xmrmx_48ef32e0-be01-49ac-ae54-32f33476eadc/insights-operator/0.log" Jul 06 16:11:15.446460 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:15.446438 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-xmrmx_48ef32e0-be01-49ac-ae54-32f33476eadc/insights-operator/1.log" Jul 06 16:11:15.479850 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:15.479825 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9s9qg_c63b1ff2-df2d-4354-9707-4bee7aa4ccb3/kube-rbac-proxy/0.log" Jul 06 16:11:15.514323 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:15.514298 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9s9qg_c63b1ff2-df2d-4354-9707-4bee7aa4ccb3/exporter/0.log" Jul 06 16:11:15.528425 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:15.528406 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-96f6w/perf-node-gather-daemonset-2ssbj" Jul 06 16:11:15.543475 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:15.543451 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9s9qg_c63b1ff2-df2d-4354-9707-4bee7aa4ccb3/extractor/0.log" Jul 06 16:11:18.296061 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:18.296030 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-nktks_0bb58fc6-57b0-4511-bb64-ca0a43d05ee0/manager/0.log" Jul 06 16:11:18.358804 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:18.358776 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vhd2b_46d03c46-7d6d-4084-8b9a-211fec671746/manager/1.log" Jul 06 16:11:18.418302 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:18.418275 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vhd2b_46d03c46-7d6d-4084-8b9a-211fec671746/manager/0.log" Jul 06 16:11:18.459361 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:18.459305 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-68d58c4db6-p9kws_3879af8c-ce8b-4b3d-a66d-490cb2bfa2b0/maas-api/0.log" Jul 06 16:11:18.497256 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:18.497228 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722545-qqmh5_97ef6680-dccc-42fa-9521-346b03544ab0/cleanup/0.log" Jul 06 16:11:18.532205 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:18.532185 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722560-skpg7_8d951172-87e6-472e-a117-9210a1a7f20a/cleanup/0.log" Jul 06 16:11:18.638586 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:18.638555 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-957b6d485-2jkhc_ecf4090d-7bf0-4bff-a45b-9f91518ce855/manager/0.log" Jul 06 16:11:18.664972 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:18.664947 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4gpw22_874c3000-ae24-4343-9385-7e7e79ccfa33/istio-proxy/0.log" Jul 06 16:11:18.836733 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:18.836697 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-5nsgf_d9acfb2a-5f27-4ead-8402-a7a9d253d5ed/istio-proxy/0.log" Jul 06 16:11:18.861989 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:18.861961 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-5nsgf_d9acfb2a-5f27-4ead-8402-a7a9d253d5ed/istio-proxy/1.log" Jul 06 16:11:18.922423 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:18.922394 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lr7lt_82d1b0f6-6ce2-46bf-acc3-61d410363776/payload-processing/0.log" Jul 06 16:11:18.967514 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:18.967483 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bdc85857f-f2l8c_36396d1c-372d-4146-94a7-59e1c7003e40/manager/0.log" Jul 06 16:11:18.981818 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:18.981795 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 16:11:18.991639 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:18.991619 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-q2lgf_6283d26a-0676-4b9e-be5b-7a41a44079ae/console-operator/1.log" Jul 06 16:11:18.994375 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:18.994355 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bdc85857f-f2l8c_36396d1c-372d-4146-94a7-59e1c7003e40/copy-manifests/0.log" Jul 06 16:11:19.031743 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:19.031721 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-kmhdx_19f75d9b-f72a-4dea-ab21-87a689803e9b/postgres/0.log" Jul 06 16:11:20.282053 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:20.282026 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4gpw22_874c3000-ae24-4343-9385-7e7e79ccfa33/istio-proxy/0.log" Jul 06 16:11:20.450400 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:20.450367 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-5nsgf_d9acfb2a-5f27-4ead-8402-a7a9d253d5ed/istio-proxy/0.log" Jul 06 16:11:20.478314 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:20.478285 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-5nsgf_d9acfb2a-5f27-4ead-8402-a7a9d253d5ed/istio-proxy/1.log" Jul 06 16:11:20.542060 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:20.541993 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lr7lt_82d1b0f6-6ce2-46bf-acc3-61d410363776/payload-processing/0.log" Jul 06 16:11:20.566514 ip-10-0-129-225 kubenswrapper[2561]: I0706 16:11:20.566488 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-df4586b48-bldnw_a4caab86-7302-4ef7-8241-7dea80d7ef13/manager/0.log"