Jul 07 14:07:21.443560 ip-10-0-138-57 systemd[1]: Starting Kubernetes Kubelet... Jul 07 14:07:21.893110 ip-10-0-138-57 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 14:07:21.893110 ip-10-0-138-57 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 14:07:21.893110 ip-10-0-138-57 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 14:07:21.893110 ip-10-0-138-57 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 14:07:21.893110 ip-10-0-138-57 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 14:07:21.896031 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.895885 2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 14:07:21.900998 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.900981 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:07:21.900998 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.900997 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:07:21.900998 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901001 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901005 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901007 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901010 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901013 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901015 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901018 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901023 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901027 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901029 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901032 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901035 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901037 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901040 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901042 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901045 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901047 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901050 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901052 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901055 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:07:21.901097 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901058 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901060 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901063 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901065 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901068 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901070 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901073 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901075 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901078 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901081 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901083 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901086 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901088 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901091 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901094 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901097 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901100 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901103 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901105 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:07:21.901624 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901108 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901111 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901113 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901116 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901119 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901121 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901124 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901126 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901137 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901139 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901142 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901144 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901147 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901149 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901152 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901155 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901157 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901159 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901162 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901164 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:07:21.902119 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901167 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901170 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901172 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901174 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901177 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901179 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901182 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901187 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901192 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901195 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901199 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901202 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901205 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901207 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901211 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901214 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901217 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901219 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901222 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:07:21.902659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901225 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901228 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901230 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901232 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901236 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901240 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901243 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901245 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901249 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901252 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901255 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901258 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901274 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901278 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901280 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901283 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901285 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901288 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.901291 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:07:21.903169 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902003 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902010 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902013 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902016 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902019 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902022 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902025 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902028 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902030 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902033 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902035 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902038 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902040 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902045 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902049 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902051 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902054 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902057 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902059 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:07:21.903665 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902062 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902064 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902067 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902069 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902072 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902074 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902077 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902080 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902082 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902085 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902089 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902092 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902095 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902097 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902100 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902103 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902105 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902108 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902110 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:07:21.904156 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902113 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902116 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902119 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902122 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902124 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902127 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902130 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902133 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902135 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902138 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902140 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902143 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902145 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902148 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902150 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902153 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902155 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902158 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902160 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902163 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:07:21.904668 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902166 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902168 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902172 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902176 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902179 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902182 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902185 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902188 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902190 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902193 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902195 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902197 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902200 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902203 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902205 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902207 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902210 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902212 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902215 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:07:21.905240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902217 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902220 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902222 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902224 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902227 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902229 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902232 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902234 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902236 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902239 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902242 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902244 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902247 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902249 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902252 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902255 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902257 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902277 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902282 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902286 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:07:21.905752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902294 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902297 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902703 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902708 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902711 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902714 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902718 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902721 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902724 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902727 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902729 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902732 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902735 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902737 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902740 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902742 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902745 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902748 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902752 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:07:21.906308 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902755 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902757 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902760 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902762 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902765 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902768 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902770 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902773 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902775 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902778 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902780 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902782 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902785 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902789 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902791 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902794 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902797 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902799 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902802 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902804 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:07:21.906794 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902808 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902810 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902812 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902815 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902817 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902820 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902822 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902825 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902827 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902830 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902832 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902835 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902837 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902840 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902842 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902844 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902847 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902849 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902852 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902854 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:07:21.907318 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902857 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902859 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902861 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902865 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902869 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902873 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902876 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902879 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902882 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902884 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902887 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902890 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902892 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902895 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902898 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902901 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902905 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902908 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902910 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:07:21.907867 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902913 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902915 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902918 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902921 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902923 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902926 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902929 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902931 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902934 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902936 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902939 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902942 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902944 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902947 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902949 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902951 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902954 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902956 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902960 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902962 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:07:21.908374 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902965 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902968 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.902970 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903036 2560 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903043 2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903049 2560 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903053 2560 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903058 2560 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903061 2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903065 2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903069 2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903072 2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903075 2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903078 2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903081 2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903084 2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903087 2560 flags.go:64] FLAG: --cgroup-root="" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903090 2560 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903093 2560 flags.go:64] FLAG: --client-ca-file="" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903096 2560 flags.go:64] FLAG: --cloud-provider="external" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903098 2560 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903102 2560 flags.go:64] FLAG: --cluster-domain="" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903106 2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 14:07:21.908883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903110 2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903112 2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903116 2560 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903120 2560 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903123 2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903126 2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903129 2560 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903134 2560 flags.go:64] FLAG: --contention-profiling="false" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903137 2560 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903139 2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903142 2560 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903145 2560 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903149 2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903152 2560 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903155 2560 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903158 2560 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903161 2560 flags.go:64] FLAG: --enable-server="true" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903164 2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903168 2560 flags.go:64] FLAG: --event-burst="100" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903172 2560 flags.go:64] FLAG: --event-qps="50" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903174 2560 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903178 2560 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903181 2560 flags.go:64] FLAG: --eviction-hard="" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903185 2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 14:07:21.909518 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903187 2560 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903190 2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903193 2560 flags.go:64] FLAG: --eviction-soft="" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903196 2560 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903199 2560 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903201 2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903204 2560 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903208 2560 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903211 2560 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903213 2560 flags.go:64] FLAG: --feature-gates="" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903217 2560 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903220 2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903223 2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903226 2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903229 2560 flags.go:64] FLAG: --healthz-port="10248" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903232 2560 flags.go:64] FLAG: --help="false" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903235 2560 flags.go:64] FLAG: --hostname-override="ip-10-0-138-57.ec2.internal" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903239 2560 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903241 2560 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903244 2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903247 2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903251 2560 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903253 2560 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 14:07:21.910148 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903256 2560 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903259 2560 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903275 2560 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903279 2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903282 2560 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903285 2560 flags.go:64] FLAG: --kube-reserved="" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903288 2560 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903291 2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903294 2560 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903296 2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903299 2560 flags.go:64] FLAG: --lock-file="" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903302 2560 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903305 2560 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903307 2560 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903312 2560 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903315 2560 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903319 2560 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903322 2560 flags.go:64] FLAG: --logging-format="text" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903324 2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903328 2560 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903330 2560 flags.go:64] FLAG: --manifest-url="" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903333 2560 flags.go:64] FLAG: --manifest-url-header="" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903338 2560 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903341 2560 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903346 2560 flags.go:64] FLAG: --max-pods="110" Jul 07 14:07:21.910783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903349 2560 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903352 2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903355 2560 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903358 2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903361 2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903363 2560 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903366 2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903374 2560 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903376 2560 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903379 2560 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903382 2560 flags.go:64] FLAG: --pod-cidr="" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903385 2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903391 2560 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903394 2560 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903397 2560 flags.go:64] FLAG: --pods-per-core="0" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903400 2560 flags.go:64] FLAG: --port="10250" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903403 2560 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903406 2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0edb9d415860335e1" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903409 2560 flags.go:64] FLAG: --qos-reserved="" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903411 2560 flags.go:64] FLAG: --read-only-port="10255" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903414 2560 flags.go:64] FLAG: --register-node="true" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903417 2560 flags.go:64] FLAG: --register-with-taints="" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903424 2560 flags.go:64] FLAG: --registry-burst="10" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903428 2560 flags.go:64] FLAG: --registry-qps="5" Jul 07 14:07:21.911446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903430 2560 flags.go:64] FLAG: --reserved-cpus="" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903433 2560 flags.go:64] FLAG: --reserved-memory="" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903437 2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903440 2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903443 2560 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903446 2560 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903449 2560 flags.go:64] FLAG: --runonce="false" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903452 2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903456 2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903459 2560 flags.go:64] FLAG: --seccomp-default="false" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903462 2560 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903465 2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903468 2560 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903471 2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903474 2560 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903476 2560 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903479 2560 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903482 2560 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903484 2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903487 2560 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903491 2560 flags.go:64] FLAG: --system-cgroups="" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903493 2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903498 2560 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903501 2560 flags.go:64] FLAG: --tls-cert-file="" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903503 2560 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 14:07:21.912054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903507 2560 flags.go:64] FLAG: --tls-min-version="" Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903509 2560 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903512 2560 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903514 2560 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903517 2560 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903520 2560 flags.go:64] FLAG: --v="2" Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903525 2560 flags.go:64] FLAG: --version="false" Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903529 2560 flags.go:64] FLAG: --vmodule="" Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903533 2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903536 2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903652 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903655 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903658 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903661 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903665 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903670 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903674 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903677 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903679 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903682 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903684 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:07:21.912718 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903687 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903689 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903692 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903694 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903697 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903699 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903702 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903705 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903707 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903709 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903712 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903715 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903718 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903720 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903722 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903725 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903727 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903731 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903734 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903736 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:07:21.913307 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903739 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903741 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903743 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903746 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903748 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903751 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903754 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903757 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903759 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903762 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903764 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903768 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903771 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903774 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903777 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903780 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903782 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903785 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903787 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903790 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:07:21.913838 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903792 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903795 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903797 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903800 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903802 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903804 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903807 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903809 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903811 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903815 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903817 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903820 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903822 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903824 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903827 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903829 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903832 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903834 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903838 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903840 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:07:21.914365 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903843 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903845 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903848 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903850 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903853 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903855 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903857 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903860 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903862 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903865 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903867 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903869 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903872 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903875 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903877 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903879 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903882 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903884 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903886 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903890 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:07:21.914885 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903893 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:07:21.915423 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903899 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:07:21.915423 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903901 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:07:21.915423 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903904 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:07:21.915423 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903907 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:07:21.915423 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903909 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:07:21.915423 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903912 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:07:21.915423 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.903915 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:07:21.915423 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.903920 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 14:07:21.915423 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.911584 2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 14:07:21.915423 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.911603 2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 14:07:21.915423 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.911622 2560 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 14:07:21.915423 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.911627 2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 14:07:21.915423 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911700 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:07:21.915423 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911705 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:07:21.915423 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911708 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911711 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911714 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911717 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911720 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911723 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911725 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911728 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911730 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911734 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911738 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911741 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911744 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911747 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911749 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911752 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911754 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911757 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911759 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:07:21.915821 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911763 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911766 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911769 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911771 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911774 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911776 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911779 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911782 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911791 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911794 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911796 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911799 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911802 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911804 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911807 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911810 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911812 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911814 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911817 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911820 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:07:21.916414 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911823 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911825 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911828 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911830 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911839 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911841 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911844 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911846 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911849 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911851 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911853 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911856 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911858 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911861 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911864 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911866 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911868 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911871 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911873 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911876 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:07:21.916925 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911879 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911887 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911889 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911892 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911895 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911897 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911899 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911902 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911904 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911907 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911909 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911912 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911914 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911917 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911919 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911922 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911924 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911926 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911929 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911931 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:07:21.917454 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911934 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911938 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911941 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911944 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911946 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911949 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911951 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911953 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911956 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911958 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911961 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911964 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911967 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911969 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911979 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911982 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911984 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:07:21.917980 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.911986 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:07:21.918427 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.911992 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 14:07:21.918427 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912149 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:07:21.918427 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912155 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:07:21.918427 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912158 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:07:21.918427 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912161 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:07:21.918427 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912163 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:07:21.918427 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912166 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:07:21.918427 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912168 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:07:21.918427 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912171 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:07:21.918427 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912173 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:07:21.918427 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912176 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:07:21.918427 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912179 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:07:21.918427 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912181 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:07:21.918427 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912184 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912187 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912190 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912193 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912195 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912197 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912200 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912204 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912207 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912210 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912213 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912216 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912219 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912221 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912224 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912227 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912235 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912238 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912240 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:07:21.918806 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912242 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912245 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912247 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912250 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912252 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912254 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912257 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912259 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912280 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912283 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912286 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912289 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912292 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912294 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912297 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912299 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912302 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912304 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912307 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912309 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:07:21.919302 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912312 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912314 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912317 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912319 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912322 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912325 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912327 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912330 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912333 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912336 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912345 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912347 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912350 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912352 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912355 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912357 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912360 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912362 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912364 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912367 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912369 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:07:21.919832 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912372 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912374 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912377 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912379 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912381 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912384 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912386 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912389 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912392 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912394 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912396 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912399 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912401 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912404 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912406 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912408 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912411 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912413 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912416 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:07:21.920404 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912418 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:07:21.920987 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912421 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:07:21.920987 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912424 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:07:21.920987 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912433 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:07:21.920987 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912436 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:07:21.920987 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912439 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:07:21.920987 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912442 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:07:21.920987 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:21.912444 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:07:21.920987 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.912449 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 14:07:21.920987 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.912587 2560 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 14:07:21.920987 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.914918 2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 14:07:21.920987 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.916992 2560 server.go:1019] "Starting client certificate rotation" Jul 07 14:07:21.920987 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.917110 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 14:07:21.920987 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.917176 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 14:07:21.921357 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.919851 2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 14:07:21.926405 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.926381 2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 14:07:21.940987 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.940963 2560 log.go:25] "Validated CRI v1 runtime API" Jul 07 14:07:21.943215 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.943201 2560 log.go:25] "Validated CRI v1 image API" Jul 07 14:07:21.944061 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.944046 2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 14:07:21.947873 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.947846 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 14:07:21.948448 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.948414 2560 fs.go:135] Filesystem UUIDs: map[6adf9db1-368f-40b2-ae71-4483e7b7372e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d2e69d3c-de82-4be5-9143-bc7e0882f9ad:/dev/nvme0n1p3] Jul 07 14:07:21.948543 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.948449 2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 14:07:21.953921 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.953793 2560 manager.go:216] Machine: {Timestamp:2026-07-07 14:07:21.952355763 +0000 UTC m=+0.384598257 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102626 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec249a8abaf161405847d73e71164a02 SystemUUID:ec249a8a-baf1-6140-5847-d73e71164a02 BootID:c0fe1ebb-5ffa-4088-8421-ae612ff0d5dd Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:26:5b:e6:a3:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:26:5b:e6:a3:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:24:22:b8:cc:99 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 14:07:21.953921 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.953911 2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 14:07:21.954085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.954019 2560 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 14:07:21.957312 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.957256 2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 14:07:21.957458 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.957316 2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-57.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 14:07:21.958109 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.958098 2560 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 14:07:21.958141 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.958112 2560 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 14:07:21.958141 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.958120 2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 14:07:21.958141 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.958135 2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 14:07:21.959681 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.959669 2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 14:07:21.962936 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.962924 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 07 14:07:21.963052 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.963042 2560 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 14:07:21.963095 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.963075 2560 kubelet.go:486] "Attempting to sync node with API server" Jul 07 14:07:21.963095 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.963085 2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 14:07:21.963163 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.963096 2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 14:07:21.963163 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.963106 2560 kubelet.go:398] "Adding apiserver pod source" Jul 07 14:07:21.963163 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.963114 2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 14:07:21.963610 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.963587 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 14:07:21.963663 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.963615 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 14:07:21.966259 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.966241 2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 14:07:21.968333 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.968320 2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 14:07:21.968395 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.968351 2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 14:07:21.969639 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.969627 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 14:07:21.969704 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.969643 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 14:07:21.969704 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.969649 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 14:07:21.969704 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.969655 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 14:07:21.969822 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.969705 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 14:07:21.969822 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.969715 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 14:07:21.969822 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.969741 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 14:07:21.969822 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.969752 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 14:07:21.969822 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.969762 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 14:07:21.969822 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.969775 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 14:07:21.969822 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.969801 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 14:07:21.969822 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.969814 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 14:07:21.970032 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.969858 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 14:07:21.970032 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.969867 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 14:07:21.974048 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.974030 2560 server.go:1268] "Started kubelet" Jul 07 14:07:21.974305 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.974249 2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 14:07:21.974376 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.974340 2560 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 14:07:21.974879 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.974813 2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 14:07:21.975066 ip-10-0-138-57 systemd[1]: Started Kubernetes Kubelet. Jul 07 14:07:21.976238 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.976211 2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 14:07:21.977145 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.977123 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6vhkm" Jul 07 14:07:21.977239 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.977182 2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-57.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 14:07:21.977503 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.977488 2560 server.go:310] "Adding debug handlers to kubelet server" Jul 07 14:07:21.978677 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:21.977863 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-57.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 14:07:21.978765 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:21.977857 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 14:07:21.981595 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:21.981569 2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 14:07:21.983524 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.983507 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6vhkm" Jul 07 14:07:21.984342 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.984329 2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 14:07:21.984406 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.984342 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 14:07:21.984491 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:21.984471 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:21.985140 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.985124 2560 factory.go:55] Registering systemd factory Jul 07 14:07:21.985213 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.985151 2560 factory.go:223] Registration of the systemd container factory successfully Jul 07 14:07:21.985213 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.985139 2560 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 14:07:21.985213 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.985169 2560 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 14:07:21.985381 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.985313 2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 14:07:21.985381 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.985321 2560 reconstruct.go:97] "Volume reconstruction finished" Jul 07 14:07:21.985381 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.985329 2560 reconciler.go:29] "Reconciler: start to sync state" Jul 07 14:07:21.985588 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.985576 2560 factory.go:153] Registering CRI-O factory Jul 07 14:07:21.985672 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.985591 2560 factory.go:223] Registration of the crio container factory successfully Jul 07 14:07:21.985672 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.985643 2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 14:07:21.985774 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.985674 2560 factory.go:103] Registering Raw factory Jul 07 14:07:21.985774 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.985689 2560 manager.go:1196] Started watching for new ooms in manager Jul 07 14:07:21.987662 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.987646 2560 manager.go:349] Starting recovery of all containers Jul 07 14:07:21.990510 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.990487 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:07:21.993739 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:21.993717 2560 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-57.ec2.internal\" not found" node="ip-10-0-138-57.ec2.internal" Jul 07 14:07:21.994089 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.994047 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 14:07:21.997757 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:21.997738 2560 manager.go:354] Recovery completed Jul 07 14:07:22.002447 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.002432 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:07:22.005370 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.005355 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:07:22.005442 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.005385 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:07:22.005442 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.005398 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:07:22.005944 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.005932 2560 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 14:07:22.006007 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.005944 2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 14:07:22.006007 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.005959 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:07:22.006007 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.005966 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 07 14:07:22.008670 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.008650 2560 policy_none.go:49] "None policy: Start" Jul 07 14:07:22.008753 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.008675 2560 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 14:07:22.008753 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.008687 2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 14:07:22.008827 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.008800 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:07:22.008874 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.008828 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:07:22.008874 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.008840 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:07:22.010020 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.010008 2560 policy_none.go:47] "Start" Jul 07 14:07:22.051283 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.051028 2560 manager.go:337] "Starting Device Plugin manager" Jul 07 14:07:22.051417 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.051303 2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 14:07:22.051417 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.051315 2560 server.go:87] "Starting device plugin registration server" Jul 07 14:07:22.051570 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.051559 2560 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 14:07:22.051621 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.051572 2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 14:07:22.051674 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.051657 2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 14:07:22.051758 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.051746 2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 14:07:22.051803 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.051759 2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 14:07:22.052332 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.052307 2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 14:07:22.052431 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.052361 2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:22.152664 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.152577 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:07:22.153773 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.153718 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:07:22.153773 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.153749 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:07:22.153773 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.153764 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:07:22.153914 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.153791 2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.163809 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.163783 2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.163809 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.163808 2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-57.ec2.internal\": node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:22.184300 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.184251 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:22.202475 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.202449 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 14:07:22.202581 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.202492 2560 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 14:07:22.202581 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.202515 2560 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 14:07:22.202581 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.202552 2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 14:07:22.205197 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.205175 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:07:22.284869 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.284833 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:22.303171 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.303147 2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-57.ec2.internal"] Jul 07 14:07:22.303311 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.303214 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:07:22.304208 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.304188 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:07:22.304320 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.304228 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:07:22.304320 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.304244 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:07:22.306008 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.305996 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:07:22.306141 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.306121 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.306193 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.306157 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:07:22.306728 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.306709 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:07:22.306821 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.306741 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:07:22.306821 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.306758 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:07:22.306821 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.306741 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:07:22.306965 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.306837 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:07:22.306965 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.306857 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:07:22.308365 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.308350 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.308445 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.308381 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:07:22.309069 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.309052 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:07:22.309143 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.309085 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:07:22.309143 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.309101 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:07:22.329952 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.329926 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-57.ec2.internal\" not found" node="ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.334388 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.334372 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-57.ec2.internal\" not found" node="ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.385942 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.385906 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:22.387087 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.387067 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9386b51d35694e977222ff73c9ef8b76-config\") pod \"kube-apiserver-proxy-ip-10-0-138-57.ec2.internal\" (UID: \"9386b51d35694e977222ff73c9ef8b76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.387142 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.387097 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf520380696161bcb2a0b72df4da4cb4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal\" (UID: \"cf520380696161bcb2a0b72df4da4cb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.387142 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.387117 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf520380696161bcb2a0b72df4da4cb4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal\" (UID: \"cf520380696161bcb2a0b72df4da4cb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.486788 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.486704 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:22.487970 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.487947 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf520380696161bcb2a0b72df4da4cb4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal\" (UID: \"cf520380696161bcb2a0b72df4da4cb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.488049 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.487984 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf520380696161bcb2a0b72df4da4cb4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal\" (UID: \"cf520380696161bcb2a0b72df4da4cb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.488049 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.488002 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9386b51d35694e977222ff73c9ef8b76-config\") pod \"kube-apiserver-proxy-ip-10-0-138-57.ec2.internal\" (UID: \"9386b51d35694e977222ff73c9ef8b76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.488145 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.488057 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9386b51d35694e977222ff73c9ef8b76-config\") pod \"kube-apiserver-proxy-ip-10-0-138-57.ec2.internal\" (UID: \"9386b51d35694e977222ff73c9ef8b76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.488145 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.488058 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cf520380696161bcb2a0b72df4da4cb4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal\" (UID: \"cf520380696161bcb2a0b72df4da4cb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.488145 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.488077 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf520380696161bcb2a0b72df4da4cb4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal\" (UID: \"cf520380696161bcb2a0b72df4da4cb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.591430 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.591377 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:22.635996 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.635961 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.638310 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.638295 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-57.ec2.internal" Jul 07 14:07:22.691497 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.691455 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:22.792495 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.792403 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:22.893497 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.893465 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:22.916362 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.916337 2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 14:07:22.916814 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.916530 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 14:07:22.916814 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.916550 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 14:07:22.985741 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.985695 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 14:02:22 +0000 UTC" deadline="2028-01-21 20:59:33.940586497 +0000 UTC" Jul 07 14:07:22.985741 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:22.985736 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13518h52m10.954855563s" Jul 07 14:07:22.994159 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:22.994128 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:23.095194 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:23.095105 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:23.195631 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:23.195599 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:23.263196 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:23.263164 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf520380696161bcb2a0b72df4da4cb4.slice/crio-e3de22575a0348e0b9a73f95f8aaeace3dd60f7dec4110500d6bfc3978190d5b WatchSource:0}: Error finding container e3de22575a0348e0b9a73f95f8aaeace3dd60f7dec4110500d6bfc3978190d5b: Status 404 returned error can't find the container with id e3de22575a0348e0b9a73f95f8aaeace3dd60f7dec4110500d6bfc3978190d5b Jul 07 14:07:23.263557 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:23.263544 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9386b51d35694e977222ff73c9ef8b76.slice/crio-32a0c5ae92fede6cde162237cdad4fc528b5937bab7045a36d938ef43747c371 WatchSource:0}: Error finding container 32a0c5ae92fede6cde162237cdad4fc528b5937bab7045a36d938ef43747c371: Status 404 returned error can't find the container with id 32a0c5ae92fede6cde162237cdad4fc528b5937bab7045a36d938ef43747c371 Jul 07 14:07:23.268527 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.268511 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:07:23.296689 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:23.296657 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:23.305528 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.305502 2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:07:23.397039 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:23.396938 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:23.497111 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:23.497055 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-57.ec2.internal\" not found" Jul 07 14:07:23.523643 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.523614 2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:07:23.589593 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.589557 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal" Jul 07 14:07:23.603104 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.603081 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 14:07:23.604254 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.604240 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-57.ec2.internal" Jul 07 14:07:23.610799 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.610781 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 14:07:23.919327 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.919301 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:07:23.966904 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.966867 2560 apiserver.go:52] "Watching apiserver" Jul 07 14:07:23.973857 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.973829 2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 14:07:23.974257 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.974236 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-57.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck","openshift-cluster-node-tuning-operator/tuned-qjlhn","openshift-dns/node-resolver-6bc6n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal","openshift-multus/multus-vqk55","openshift-network-operator/iptables-alerter-d4tt5","openshift-ovn-kubernetes/ovnkube-node-wprk2","kube-system/global-pull-secret-syncer-xhkww","kube-system/konnectivity-agent-9wbnm","openshift-image-registry/node-ca-5kt5s","openshift-multus/multus-additional-cni-plugins-bpdnk","openshift-multus/network-metrics-daemon-8rf2w","openshift-network-diagnostics/network-check-target-vsfxs"] Jul 07 14:07:23.976691 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.976672 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:23.976783 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:23.976759 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:23.977858 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.977841 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d4tt5" Jul 07 14:07:23.979052 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.979025 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:23.980234 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.980216 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-76jk8\"" Jul 07 14:07:23.980322 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.980298 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:07:23.980415 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.980324 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6bc6n" Jul 07 14:07:23.980415 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.980381 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 14:07:23.980528 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.980487 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 14:07:23.981170 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.981149 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:07:23.981253 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.981182 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 14:07:23.981253 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.981154 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hp492\"" Jul 07 14:07:23.981706 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.981691 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vqk55" Jul 07 14:07:23.982443 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.982421 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 14:07:23.982589 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.982572 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 14:07:23.982708 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.982691 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sw88v\"" Jul 07 14:07:23.983075 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.983057 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:23.983808 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.983789 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 14:07:23.983911 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.983818 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fvc6h\"" Jul 07 14:07:23.983974 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.983938 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 14:07:23.984021 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.984009 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 14:07:23.984076 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.984061 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 14:07:23.984522 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.984496 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 14:07:23.984634 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.984591 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:23.984721 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:23.984701 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:23.985543 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.985520 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 14:07:23.985644 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.985548 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2474k\"" Jul 07 14:07:23.985951 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.985929 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 14:07:23.986058 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.986038 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 14:07:23.986358 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.986316 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 14:07:23.986480 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.986463 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 14:07:23.986543 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.986471 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9wbnm" Jul 07 14:07:23.986543 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.986536 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 14:07:23.988153 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.988136 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5kt5s" Jul 07 14:07:23.988873 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.988853 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lk78f\"" Jul 07 14:07:23.988993 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.988970 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 14:07:23.989316 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.989303 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 14:07:23.990029 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.990006 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:23.991366 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.991315 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 14:07:23.991672 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.991654 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7jlfm\"" Jul 07 14:07:23.991762 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.991711 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 14:07:23.992131 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.992114 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:23.992327 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:23.992303 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:23.993913 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.992474 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 14:07:23.993913 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.992699 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d95kk\"" Jul 07 14:07:23.993913 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.992706 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 14:07:23.993913 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.992821 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 14:07:23.995205 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.995180 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:23.997557 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.997539 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lvxfx\"" Jul 07 14:07:23.997716 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.997700 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 14:07:23.997793 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.997705 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 14:07:23.997855 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.997794 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 14:07:23.997855 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:23.997836 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 14:07:24.000384 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000365 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.000475 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000416 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jfwq\" (UniqueName: \"kubernetes.io/projected/d0a6823a-f90b-4bdf-9682-a00e462d88bb-kube-api-access-4jfwq\") pod \"iptables-alerter-d4tt5\" (UID: \"d0a6823a-f90b-4bdf-9682-a00e462d88bb\") " pod="openshift-network-operator/iptables-alerter-d4tt5" Jul 07 14:07:24.000475 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000443 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e80423b-0747-4914-a00d-19bdf951c3ca-tmp\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.000475 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000469 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-system-cni-dir\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.000603 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000494 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7gr6\" (UniqueName: \"kubernetes.io/projected/1ac773f2-d330-4140-97aa-8ac19ee4d555-kube-api-access-k7gr6\") pod \"node-resolver-6bc6n\" (UID: \"1ac773f2-d330-4140-97aa-8ac19ee4d555\") " pod="openshift-dns/node-resolver-6bc6n" Jul 07 14:07:24.000603 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000525 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-kubelet\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.000603 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000551 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e36c76a-c29f-42bc-9e51-4516e19d345a-ovnkube-config\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.000603 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000586 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-kubernetes\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.000744 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000615 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e36c76a-c29f-42bc-9e51-4516e19d345a-ovnkube-script-lib\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.000744 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000633 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0a6823a-f90b-4bdf-9682-a00e462d88bb-host-slash\") pod \"iptables-alerter-d4tt5\" (UID: \"d0a6823a-f90b-4bdf-9682-a00e462d88bb\") " pod="openshift-network-operator/iptables-alerter-d4tt5" Jul 07 14:07:24.000744 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000652 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-multus-conf-dir\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.000744 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000677 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-var-lib-cni-multus\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.000744 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000700 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-run-systemd\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.000744 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000726 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-etc-kubernetes\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.000984 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000771 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfj5d\" (UniqueName: \"kubernetes.io/projected/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-kube-api-access-kfj5d\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.000984 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000808 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ac773f2-d330-4140-97aa-8ac19ee4d555-hosts-file\") pod \"node-resolver-6bc6n\" (UID: \"1ac773f2-d330-4140-97aa-8ac19ee4d555\") " pod="openshift-dns/node-resolver-6bc6n" Jul 07 14:07:24.000984 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000827 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-log-socket\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.000984 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000843 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b457606-cff0-47a1-8938-778f59dd934d-dbus\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:24.000984 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000859 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc2a0c37-b521-42e0-9e34-bcdc75a54636-konnectivity-ca\") pod \"konnectivity-agent-9wbnm\" (UID: \"dc2a0c37-b521-42e0-9e34-bcdc75a54636\") " pod="kube-system/konnectivity-agent-9wbnm" Jul 07 14:07:24.000984 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000886 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-sysctl-conf\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.000984 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000917 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-var-lib-kubelet\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.000984 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000944 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-cnibin\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.000984 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000966 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ac773f2-d330-4140-97aa-8ac19ee4d555-tmp-dir\") pod \"node-resolver-6bc6n\" (UID: \"1ac773f2-d330-4140-97aa-8ac19ee4d555\") " pod="openshift-dns/node-resolver-6bc6n" Jul 07 14:07:24.000984 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.000987 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-run-ovn\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.001299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001004 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-sysconfig\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.001299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001018 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-host\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.001299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001063 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.001299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001081 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-multus-cni-dir\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.001299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001097 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-var-lib-kubelet\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.001299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001121 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-multus-daemon-config\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.001299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001146 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtlx2\" (UniqueName: \"kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2\") pod \"network-check-target-vsfxs\" (UID: \"2c345a9e-d35d-40b1-91c5-553c42d60e8f\") " pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:24.001299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001169 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-systemd-units\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.001299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001185 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-etc-openvswitch\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.001299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001201 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e36c76a-c29f-42bc-9e51-4516e19d345a-ovn-node-metrics-cert\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.001299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001220 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0a6823a-f90b-4bdf-9682-a00e462d88bb-iptables-alerter-script\") pod \"iptables-alerter-d4tt5\" (UID: \"d0a6823a-f90b-4bdf-9682-a00e462d88bb\") " pod="openshift-network-operator/iptables-alerter-d4tt5" Jul 07 14:07:24.001299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001239 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-sysctl-d\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.001299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001254 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:24.001299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001273 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-var-lib-openvswitch\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001302 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b457606-cff0-47a1-8938-778f59dd934d-kubelet-config\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001329 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc2a0c37-b521-42e0-9e34-bcdc75a54636-agent-certs\") pod \"konnectivity-agent-9wbnm\" (UID: \"dc2a0c37-b521-42e0-9e34-bcdc75a54636\") " pod="kube-system/konnectivity-agent-9wbnm" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001383 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-lib-modules\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001423 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-run-netns\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001452 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-run-openvswitch\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001483 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsdvf\" (UniqueName: \"kubernetes.io/projected/7e80423b-0747-4914-a00d-19bdf951c3ca-kube-api-access-gsdvf\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001518 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-run-netns\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001544 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-node-log\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001562 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-modprobe-d\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001588 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-sys\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001609 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-hostroot\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001636 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-cni-bin\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001667 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-multus-socket-dir-parent\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001687 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-run-multus-certs\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001714 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-slash\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001736 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp8xm\" (UniqueName: \"kubernetes.io/projected/7e36c76a-c29f-42bc-9e51-4516e19d345a-kube-api-access-zp8xm\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.001751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001756 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-tuned\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.002310 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001796 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-os-release\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.002310 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001837 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-cni-binary-copy\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.002310 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001868 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-run-k8s-cni-cncf-io\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.002310 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001891 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-cni-netd\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.002310 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001910 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e36c76a-c29f-42bc-9e51-4516e19d345a-env-overrides\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.002310 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001935 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-systemd\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.002310 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001951 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-run\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.002310 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.001966 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-var-lib-cni-bin\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.019712 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.019691 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zn795" Jul 07 14:07:24.029424 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.029405 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zn795" Jul 07 14:07:24.086476 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.086443 2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 14:07:24.102281 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102224 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsdvf\" (UniqueName: \"kubernetes.io/projected/7e80423b-0747-4914-a00d-19bdf951c3ca-kube-api-access-gsdvf\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.102281 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102269 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-run-netns\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.102523 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102305 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-node-log\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.102523 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102328 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cf123ea-2a30-4535-971f-0833cafaedea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.102523 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102355 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znpxf\" (UniqueName: \"kubernetes.io/projected/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-kube-api-access-znpxf\") pod \"network-metrics-daemon-8rf2w\" (UID: \"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603\") " pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:24.102523 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102376 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-socket-dir\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.102523 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102403 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-modprobe-d\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.102523 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102420 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-sys\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.102523 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102436 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-hostroot\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.102523 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102442 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-node-log\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.102523 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102452 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-cni-bin\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.102523 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102468 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cf123ea-2a30-4535-971f-0833cafaedea-os-release\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.102523 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102484 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cf123ea-2a30-4535-971f-0833cafaedea-cni-binary-copy\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.102523 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102500 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttl8v\" (UniqueName: \"kubernetes.io/projected/6cf123ea-2a30-4535-971f-0833cafaedea-kube-api-access-ttl8v\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.102523 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102516 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-device-dir\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102533 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-multus-socket-dir-parent\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102558 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-run-multus-certs\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102579 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-slash\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102584 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-modprobe-d\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102594 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp8xm\" (UniqueName: \"kubernetes.io/projected/7e36c76a-c29f-42bc-9e51-4516e19d345a-kube-api-access-zp8xm\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102610 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-tuned\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102641 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-os-release\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102674 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-cni-binary-copy\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102705 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-run-k8s-cni-cncf-io\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102734 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-cni-netd\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102759 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e36c76a-c29f-42bc-9e51-4516e19d345a-env-overrides\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102786 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1053d78f-a5ae-4acd-9a99-32beb279bbb4-host\") pod \"node-ca-5kt5s\" (UID: \"1053d78f-a5ae-4acd-9a99-32beb279bbb4\") " pod="openshift-image-registry/node-ca-5kt5s" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102814 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88jsd\" (UniqueName: \"kubernetes.io/projected/1053d78f-a5ae-4acd-9a99-32beb279bbb4-kube-api-access-88jsd\") pod \"node-ca-5kt5s\" (UID: \"1053d78f-a5ae-4acd-9a99-32beb279bbb4\") " pod="openshift-image-registry/node-ca-5kt5s" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102820 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-os-release\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102377 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-run-netns\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102841 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-systemd\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102903 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-systemd\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.103168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102909 2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102945 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-slash\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102959 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-sys\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102965 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-run\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102987 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-var-lib-cni-bin\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.103006 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.102996 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-hostroot\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.103038 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.103039 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jfwq\" (UniqueName: \"kubernetes.io/projected/d0a6823a-f90b-4bdf-9682-a00e462d88bb-kube-api-access-4jfwq\") pod \"iptables-alerter-d4tt5\" (UID: \"d0a6823a-f90b-4bdf-9682-a00e462d88bb\") " pod="openshift-network-operator/iptables-alerter-d4tt5" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.103067 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e80423b-0747-4914-a00d-19bdf951c3ca-tmp\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.103121 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-run\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.103139 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-system-cni-dir\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.103146 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-var-lib-cni-bin\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.103168 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7gr6\" (UniqueName: \"kubernetes.io/projected/1ac773f2-d330-4140-97aa-8ac19ee4d555-kube-api-access-k7gr6\") pod \"node-resolver-6bc6n\" (UID: \"1ac773f2-d330-4140-97aa-8ac19ee4d555\") " pod="openshift-dns/node-resolver-6bc6n" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.103177 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-run-k8s-cni-cncf-io\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.103743 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-cni-netd\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.103788 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-run-multus-certs\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.104041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.103842 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-multus-socket-dir-parent\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104084 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-system-cni-dir\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104151 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-kubelet\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104185 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e36c76a-c29f-42bc-9e51-4516e19d345a-ovnkube-config\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104214 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs\") pod \"network-metrics-daemon-8rf2w\" (UID: \"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603\") " pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104243 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-kubernetes\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104281 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e36c76a-c29f-42bc-9e51-4516e19d345a-ovnkube-script-lib\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104398 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6cf123ea-2a30-4535-971f-0833cafaedea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104482 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltbmj\" (UniqueName: \"kubernetes.io/projected/2d177a24-96d5-4871-a4c9-e9d031ea97cc-kube-api-access-ltbmj\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104513 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0a6823a-f90b-4bdf-9682-a00e462d88bb-host-slash\") pod \"iptables-alerter-d4tt5\" (UID: \"d0a6823a-f90b-4bdf-9682-a00e462d88bb\") " pod="openshift-network-operator/iptables-alerter-d4tt5" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104542 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-multus-conf-dir\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104581 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-var-lib-cni-multus\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104608 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-run-systemd\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104637 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6cf123ea-2a30-4535-971f-0833cafaedea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104672 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-etc-kubernetes\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104700 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfj5d\" (UniqueName: \"kubernetes.io/projected/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-kube-api-access-kfj5d\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104729 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ac773f2-d330-4140-97aa-8ac19ee4d555-hosts-file\") pod \"node-resolver-6bc6n\" (UID: \"1ac773f2-d330-4140-97aa-8ac19ee4d555\") " pod="openshift-dns/node-resolver-6bc6n" Jul 07 14:07:24.104919 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104755 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-log-socket\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104791 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b457606-cff0-47a1-8938-778f59dd934d-dbus\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104833 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cf123ea-2a30-4535-971f-0833cafaedea-system-cni-dir\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104944 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1053d78f-a5ae-4acd-9a99-32beb279bbb4-serviceca\") pod \"node-ca-5kt5s\" (UID: \"1053d78f-a5ae-4acd-9a99-32beb279bbb4\") " pod="openshift-image-registry/node-ca-5kt5s" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.104973 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105002 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc2a0c37-b521-42e0-9e34-bcdc75a54636-konnectivity-ca\") pod \"konnectivity-agent-9wbnm\" (UID: \"dc2a0c37-b521-42e0-9e34-bcdc75a54636\") " pod="kube-system/konnectivity-agent-9wbnm" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105032 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-sysctl-conf\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105059 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-var-lib-kubelet\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105085 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-cnibin\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105118 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ac773f2-d330-4140-97aa-8ac19ee4d555-tmp-dir\") pod \"node-resolver-6bc6n\" (UID: \"1ac773f2-d330-4140-97aa-8ac19ee4d555\") " pod="openshift-dns/node-resolver-6bc6n" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105145 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-run-ovn\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105174 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-sysconfig\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105204 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-host\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105233 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105260 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105330 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-registration-dir\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105407 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-multus-cni-dir\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.105850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105436 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-var-lib-kubelet\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105602 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-multus-daemon-config\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105639 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtlx2\" (UniqueName: \"kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2\") pod \"network-check-target-vsfxs\" (UID: \"2c345a9e-d35d-40b1-91c5-553c42d60e8f\") " pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105666 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-systemd-units\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105696 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-etc-openvswitch\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105729 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e36c76a-c29f-42bc-9e51-4516e19d345a-ovn-node-metrics-cert\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105757 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0a6823a-f90b-4bdf-9682-a00e462d88bb-iptables-alerter-script\") pod \"iptables-alerter-d4tt5\" (UID: \"d0a6823a-f90b-4bdf-9682-a00e462d88bb\") " pod="openshift-network-operator/iptables-alerter-d4tt5" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.105787 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-sysctl-d\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.106081 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.106113 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cf123ea-2a30-4535-971f-0833cafaedea-cnibin\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.106140 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-sys-fs\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.106170 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-var-lib-openvswitch\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.106224 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b457606-cff0-47a1-8938-778f59dd934d-kubelet-config\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.106252 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc2a0c37-b521-42e0-9e34-bcdc75a54636-agent-certs\") pod \"konnectivity-agent-9wbnm\" (UID: \"dc2a0c37-b521-42e0-9e34-bcdc75a54636\") " pod="kube-system/konnectivity-agent-9wbnm" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.106280 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-lib-modules\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.106308 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-run-netns\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.106354 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-run-openvswitch\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.106695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.106649 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-sysctl-d\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.107571 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.106694 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-kubelet\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.107571 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.107141 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e80423b-0747-4914-a00d-19bdf951c3ca-tmp\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.107571 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.107196 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-cni-bin\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.107571 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.107318 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e36c76a-c29f-42bc-9e51-4516e19d345a-ovnkube-config\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.107571 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.107372 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-tuned\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.107571 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.107424 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-var-lib-cni-multus\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.107571 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.107440 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-kubernetes\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.107571 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.107468 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-run-systemd\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.107571 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.107519 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-etc-kubernetes\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.107571 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.107540 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e36c76a-c29f-42bc-9e51-4516e19d345a-env-overrides\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.108065 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.107772 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ac773f2-d330-4140-97aa-8ac19ee4d555-hosts-file\") pod \"node-resolver-6bc6n\" (UID: \"1ac773f2-d330-4140-97aa-8ac19ee4d555\") " pod="openshift-dns/node-resolver-6bc6n" Jul 07 14:07:24.108065 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.107814 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-log-socket\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.108065 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.107923 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e36c76a-c29f-42bc-9e51-4516e19d345a-ovnkube-script-lib\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.108065 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.107913 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b457606-cff0-47a1-8938-778f59dd934d-dbus\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:24.108065 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.107976 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0a6823a-f90b-4bdf-9682-a00e462d88bb-host-slash\") pod \"iptables-alerter-d4tt5\" (UID: \"d0a6823a-f90b-4bdf-9682-a00e462d88bb\") " pod="openshift-network-operator/iptables-alerter-d4tt5" Jul 07 14:07:24.108065 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.107998 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-cni-binary-copy\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.108065 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108031 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-multus-conf-dir\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.108434 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108133 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-sysctl-conf\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.108434 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108177 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-var-lib-kubelet\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.108434 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108217 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-cnibin\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.108607 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108533 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ac773f2-d330-4140-97aa-8ac19ee4d555-tmp-dir\") pod \"node-resolver-6bc6n\" (UID: \"1ac773f2-d330-4140-97aa-8ac19ee4d555\") " pod="openshift-dns/node-resolver-6bc6n" Jul 07 14:07:24.108607 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108586 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-run-ovn\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.108713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108628 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-etc-sysconfig\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.108713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108669 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-host\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.108713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108671 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc2a0c37-b521-42e0-9e34-bcdc75a54636-konnectivity-ca\") pod \"konnectivity-agent-9wbnm\" (UID: \"dc2a0c37-b521-42e0-9e34-bcdc75a54636\") " pod="kube-system/konnectivity-agent-9wbnm" Jul 07 14:07:24.108868 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108718 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.108868 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108746 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-systemd-units\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.108868 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108764 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-var-lib-openvswitch\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.108868 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108775 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-etc-openvswitch\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.108868 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108783 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b457606-cff0-47a1-8938-778f59dd934d-kubelet-config\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:24.108868 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108822 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-multus-cni-dir\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.109161 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108881 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-run-openvswitch\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.109161 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.108955 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e80423b-0747-4914-a00d-19bdf951c3ca-lib-modules\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.109161 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.109032 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:07:24.109312 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.109214 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret podName:2b457606-cff0-47a1-8938-778f59dd934d nodeName:}" failed. No retries permitted until 2026-07-07 14:07:24.609186942 +0000 UTC m=+3.041429445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret") pod "global-pull-secret-syncer-xhkww" (UID: "2b457606-cff0-47a1-8938-778f59dd934d") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:07:24.109411 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.109390 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0a6823a-f90b-4bdf-9682-a00e462d88bb-iptables-alerter-script\") pod \"iptables-alerter-d4tt5\" (UID: \"d0a6823a-f90b-4bdf-9682-a00e462d88bb\") " pod="openshift-network-operator/iptables-alerter-d4tt5" Jul 07 14:07:24.109519 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.109033 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-host-var-lib-kubelet\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.109872 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.109818 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e36c76a-c29f-42bc-9e51-4516e19d345a-host-run-netns\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.109872 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.109845 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-multus-daemon-config\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.112319 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.112298 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e36c76a-c29f-42bc-9e51-4516e19d345a-ovn-node-metrics-cert\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.112476 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.112452 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc2a0c37-b521-42e0-9e34-bcdc75a54636-agent-certs\") pod \"konnectivity-agent-9wbnm\" (UID: \"dc2a0c37-b521-42e0-9e34-bcdc75a54636\") " pod="kube-system/konnectivity-agent-9wbnm" Jul 07 14:07:24.115075 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.115051 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:07:24.115175 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.115085 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:07:24.115175 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.115100 2560 projected.go:196] Error preparing data for projected volume kube-api-access-wtlx2 for pod openshift-network-diagnostics/network-check-target-vsfxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:07:24.115296 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.115218 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2 podName:2c345a9e-d35d-40b1-91c5-553c42d60e8f nodeName:}" failed. No retries permitted until 2026-07-07 14:07:24.615198008 +0000 UTC m=+3.047440509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wtlx2" (UniqueName: "kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2") pod "network-check-target-vsfxs" (UID: "2c345a9e-d35d-40b1-91c5-553c42d60e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:07:24.115699 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.115673 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jfwq\" (UniqueName: \"kubernetes.io/projected/d0a6823a-f90b-4bdf-9682-a00e462d88bb-kube-api-access-4jfwq\") pod \"iptables-alerter-d4tt5\" (UID: \"d0a6823a-f90b-4bdf-9682-a00e462d88bb\") " pod="openshift-network-operator/iptables-alerter-d4tt5" Jul 07 14:07:24.115890 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.115712 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7gr6\" (UniqueName: \"kubernetes.io/projected/1ac773f2-d330-4140-97aa-8ac19ee4d555-kube-api-access-k7gr6\") pod \"node-resolver-6bc6n\" (UID: \"1ac773f2-d330-4140-97aa-8ac19ee4d555\") " pod="openshift-dns/node-resolver-6bc6n" Jul 07 14:07:24.117170 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.117144 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp8xm\" (UniqueName: \"kubernetes.io/projected/7e36c76a-c29f-42bc-9e51-4516e19d345a-kube-api-access-zp8xm\") pod \"ovnkube-node-wprk2\" (UID: \"7e36c76a-c29f-42bc-9e51-4516e19d345a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.118398 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.118377 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfj5d\" (UniqueName: \"kubernetes.io/projected/79313367-388a-4b7a-9a7c-88cf9aeb1d2a-kube-api-access-kfj5d\") pod \"multus-vqk55\" (UID: \"79313367-388a-4b7a-9a7c-88cf9aeb1d2a\") " pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.119044 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.119014 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsdvf\" (UniqueName: \"kubernetes.io/projected/7e80423b-0747-4914-a00d-19bdf951c3ca-kube-api-access-gsdvf\") pod \"tuned-qjlhn\" (UID: \"7e80423b-0747-4914-a00d-19bdf951c3ca\") " pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.206757 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206677 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cf123ea-2a30-4535-971f-0833cafaedea-cnibin\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.206757 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206711 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-sys-fs\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.206757 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206736 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cf123ea-2a30-4535-971f-0833cafaedea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.206757 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206755 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znpxf\" (UniqueName: \"kubernetes.io/projected/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-kube-api-access-znpxf\") pod \"network-metrics-daemon-8rf2w\" (UID: \"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603\") " pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206773 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-socket-dir\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206792 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cf123ea-2a30-4535-971f-0833cafaedea-os-release\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206808 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cf123ea-2a30-4535-971f-0833cafaedea-cni-binary-copy\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206825 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttl8v\" (UniqueName: \"kubernetes.io/projected/6cf123ea-2a30-4535-971f-0833cafaedea-kube-api-access-ttl8v\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206843 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-device-dir\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206863 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1053d78f-a5ae-4acd-9a99-32beb279bbb4-host\") pod \"node-ca-5kt5s\" (UID: \"1053d78f-a5ae-4acd-9a99-32beb279bbb4\") " pod="openshift-image-registry/node-ca-5kt5s" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206879 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88jsd\" (UniqueName: \"kubernetes.io/projected/1053d78f-a5ae-4acd-9a99-32beb279bbb4-kube-api-access-88jsd\") pod \"node-ca-5kt5s\" (UID: \"1053d78f-a5ae-4acd-9a99-32beb279bbb4\") " pod="openshift-image-registry/node-ca-5kt5s" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206901 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs\") pod \"network-metrics-daemon-8rf2w\" (UID: \"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603\") " pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206922 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6cf123ea-2a30-4535-971f-0833cafaedea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206941 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltbmj\" (UniqueName: \"kubernetes.io/projected/2d177a24-96d5-4871-a4c9-e9d031ea97cc-kube-api-access-ltbmj\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206959 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6cf123ea-2a30-4535-971f-0833cafaedea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.206986 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cf123ea-2a30-4535-971f-0833cafaedea-system-cni-dir\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.207004 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1053d78f-a5ae-4acd-9a99-32beb279bbb4-serviceca\") pod \"node-ca-5kt5s\" (UID: \"1053d78f-a5ae-4acd-9a99-32beb279bbb4\") " pod="openshift-image-registry/node-ca-5kt5s" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.207020 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.207041 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.207085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.207057 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-registration-dir\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.207897 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.207160 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-registration-dir\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.207897 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.207199 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6cf123ea-2a30-4535-971f-0833cafaedea-cnibin\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.207897 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.207232 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-sys-fs\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.207897 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.207403 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6cf123ea-2a30-4535-971f-0833cafaedea-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.207897 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.207823 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-socket-dir\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.208095 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.207902 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6cf123ea-2a30-4535-971f-0833cafaedea-os-release\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.208476 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.208450 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6cf123ea-2a30-4535-971f-0833cafaedea-cni-binary-copy\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.208802 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.208777 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-device-dir\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.208886 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.208838 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1053d78f-a5ae-4acd-9a99-32beb279bbb4-host\") pod \"node-ca-5kt5s\" (UID: \"1053d78f-a5ae-4acd-9a99-32beb279bbb4\") " pod="openshift-image-registry/node-ca-5kt5s" Jul 07 14:07:24.209130 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.209109 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:07:24.209204 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.209188 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs podName:3d5ba7ef-4b44-4c07-9dc0-77a3e4223603 nodeName:}" failed. No retries permitted until 2026-07-07 14:07:24.709169132 +0000 UTC m=+3.141411627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs") pod "network-metrics-daemon-8rf2w" (UID: "3d5ba7ef-4b44-4c07-9dc0-77a3e4223603") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:07:24.209833 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.209808 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6cf123ea-2a30-4535-971f-0833cafaedea-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.210245 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.210222 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.210341 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.210304 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6cf123ea-2a30-4535-971f-0833cafaedea-system-cni-dir\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.210561 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.210536 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6cf123ea-2a30-4535-971f-0833cafaedea-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.210640 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.210625 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2d177a24-96d5-4871-a4c9-e9d031ea97cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.210754 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.210734 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1053d78f-a5ae-4acd-9a99-32beb279bbb4-serviceca\") pod \"node-ca-5kt5s\" (UID: \"1053d78f-a5ae-4acd-9a99-32beb279bbb4\") " pod="openshift-image-registry/node-ca-5kt5s" Jul 07 14:07:24.215241 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.215186 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal" event={"ID":"cf520380696161bcb2a0b72df4da4cb4","Type":"ContainerStarted","Data":"e3de22575a0348e0b9a73f95f8aaeace3dd60f7dec4110500d6bfc3978190d5b"} Jul 07 14:07:24.218408 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.218378 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-57.ec2.internal" event={"ID":"9386b51d35694e977222ff73c9ef8b76","Type":"ContainerStarted","Data":"32a0c5ae92fede6cde162237cdad4fc528b5937bab7045a36d938ef43747c371"} Jul 07 14:07:24.225219 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.225002 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88jsd\" (UniqueName: \"kubernetes.io/projected/1053d78f-a5ae-4acd-9a99-32beb279bbb4-kube-api-access-88jsd\") pod \"node-ca-5kt5s\" (UID: \"1053d78f-a5ae-4acd-9a99-32beb279bbb4\") " pod="openshift-image-registry/node-ca-5kt5s" Jul 07 14:07:24.231171 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.231096 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltbmj\" (UniqueName: \"kubernetes.io/projected/2d177a24-96d5-4871-a4c9-e9d031ea97cc-kube-api-access-ltbmj\") pod \"aws-ebs-csi-driver-node-kpgck\" (UID: \"2d177a24-96d5-4871-a4c9-e9d031ea97cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.231662 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.231636 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znpxf\" (UniqueName: \"kubernetes.io/projected/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-kube-api-access-znpxf\") pod \"network-metrics-daemon-8rf2w\" (UID: \"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603\") " pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:24.232660 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.232635 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttl8v\" (UniqueName: \"kubernetes.io/projected/6cf123ea-2a30-4535-971f-0833cafaedea-kube-api-access-ttl8v\") pod \"multus-additional-cni-plugins-bpdnk\" (UID: \"6cf123ea-2a30-4535-971f-0833cafaedea\") " pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.276060 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.276021 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:07:24.290647 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.290597 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d4tt5" Jul 07 14:07:24.300505 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.300476 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" Jul 07 14:07:24.304476 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:24.304438 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a6823a_f90b_4bdf_9682_a00e462d88bb.slice/crio-6e09be7022d05a8ed9ef461581abc289b49165f6559f2b4b826b074e8abbac73 WatchSource:0}: Error finding container 6e09be7022d05a8ed9ef461581abc289b49165f6559f2b4b826b074e8abbac73: Status 404 returned error can't find the container with id 6e09be7022d05a8ed9ef461581abc289b49165f6559f2b4b826b074e8abbac73 Jul 07 14:07:24.307866 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.306996 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6bc6n" Jul 07 14:07:24.311702 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.311682 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vqk55" Jul 07 14:07:24.312084 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:24.312051 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e80423b_0747_4914_a00d_19bdf951c3ca.slice/crio-9b3ec987f4ff964cad7c06b5f7e40549095f8952a248565d556b136abcbf5efe WatchSource:0}: Error finding container 9b3ec987f4ff964cad7c06b5f7e40549095f8952a248565d556b136abcbf5efe: Status 404 returned error can't find the container with id 9b3ec987f4ff964cad7c06b5f7e40549095f8952a248565d556b136abcbf5efe Jul 07 14:07:24.317991 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.317963 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:24.323621 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.323447 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9wbnm" Jul 07 14:07:24.323621 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:24.323492 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac773f2_d330_4140_97aa_8ac19ee4d555.slice/crio-82208e3a0e35ab626c3f2ac2061cd9ef3c188145d9079f015f33752e049d5090 WatchSource:0}: Error finding container 82208e3a0e35ab626c3f2ac2061cd9ef3c188145d9079f015f33752e049d5090: Status 404 returned error can't find the container with id 82208e3a0e35ab626c3f2ac2061cd9ef3c188145d9079f015f33752e049d5090 Jul 07 14:07:24.327358 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:24.327305 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79313367_388a_4b7a_9a7c_88cf9aeb1d2a.slice/crio-6332ceb1cf251725b68149d0dda02e60dd7f1b1d511584c560c94d95e09268cf WatchSource:0}: Error finding container 6332ceb1cf251725b68149d0dda02e60dd7f1b1d511584c560c94d95e09268cf: Status 404 returned error can't find the container with id 6332ceb1cf251725b68149d0dda02e60dd7f1b1d511584c560c94d95e09268cf Jul 07 14:07:24.330538 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.330515 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5kt5s" Jul 07 14:07:24.335044 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.334901 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bpdnk" Jul 07 14:07:24.336781 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:24.336733 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e36c76a_c29f_42bc_9e51_4516e19d345a.slice/crio-ba79e5e9cb25a4233c2a8e882cd07c095e4a1111f0627b6c3b3f1e36d135a80a WatchSource:0}: Error finding container ba79e5e9cb25a4233c2a8e882cd07c095e4a1111f0627b6c3b3f1e36d135a80a: Status 404 returned error can't find the container with id ba79e5e9cb25a4233c2a8e882cd07c095e4a1111f0627b6c3b3f1e36d135a80a Jul 07 14:07:24.341237 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.340558 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" Jul 07 14:07:24.356024 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:24.355991 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2a0c37_b521_42e0_9e34_bcdc75a54636.slice/crio-a7ab9aec206d424dc54e84a699cff73569ac5a94931507afa998cd98696f50af WatchSource:0}: Error finding container a7ab9aec206d424dc54e84a699cff73569ac5a94931507afa998cd98696f50af: Status 404 returned error can't find the container with id a7ab9aec206d424dc54e84a699cff73569ac5a94931507afa998cd98696f50af Jul 07 14:07:24.363873 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:24.363849 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d177a24_96d5_4871_a4c9_e9d031ea97cc.slice/crio-230f6a94a84b41c59efe363ae78bfb7e88e06e6db255490b39b6115bbd0c992e WatchSource:0}: Error finding container 230f6a94a84b41c59efe363ae78bfb7e88e06e6db255490b39b6115bbd0c992e: Status 404 returned error can't find the container with id 230f6a94a84b41c59efe363ae78bfb7e88e06e6db255490b39b6115bbd0c992e Jul 07 14:07:24.610196 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.610155 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:24.610396 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.610337 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:07:24.610396 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.610395 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret podName:2b457606-cff0-47a1-8938-778f59dd934d nodeName:}" failed. No retries permitted until 2026-07-07 14:07:25.610380319 +0000 UTC m=+4.042622801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret") pod "global-pull-secret-syncer-xhkww" (UID: "2b457606-cff0-47a1-8938-778f59dd934d") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:07:24.710900 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.710861 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtlx2\" (UniqueName: \"kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2\") pod \"network-check-target-vsfxs\" (UID: \"2c345a9e-d35d-40b1-91c5-553c42d60e8f\") " pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:24.711072 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:24.710948 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs\") pod \"network-metrics-daemon-8rf2w\" (UID: \"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603\") " pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:24.711072 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.711046 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:07:24.711072 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.711048 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:07:24.711072 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.711073 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:07:24.711238 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.711086 2560 projected.go:196] Error preparing data for projected volume kube-api-access-wtlx2 for pod openshift-network-diagnostics/network-check-target-vsfxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:07:24.711238 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.711117 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs podName:3d5ba7ef-4b44-4c07-9dc0-77a3e4223603 nodeName:}" failed. No retries permitted until 2026-07-07 14:07:25.711098661 +0000 UTC m=+4.143341163 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs") pod "network-metrics-daemon-8rf2w" (UID: "3d5ba7ef-4b44-4c07-9dc0-77a3e4223603") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:07:24.711238 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:24.711135 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2 podName:2c345a9e-d35d-40b1-91c5-553c42d60e8f nodeName:}" failed. No retries permitted until 2026-07-07 14:07:25.711125664 +0000 UTC m=+4.143368148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wtlx2" (UniqueName: "kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2") pod "network-check-target-vsfxs" (UID: "2c345a9e-d35d-40b1-91c5-553c42d60e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:07:25.032312 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:25.032178 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:02:24 +0000 UTC" deadline="2028-04-16 00:29:37.09228694 +0000 UTC" Jul 07 14:07:25.032312 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:25.032246 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15562h22m12.06004544s" Jul 07 14:07:25.222359 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:25.222316 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vqk55" event={"ID":"79313367-388a-4b7a-9a7c-88cf9aeb1d2a","Type":"ContainerStarted","Data":"6332ceb1cf251725b68149d0dda02e60dd7f1b1d511584c560c94d95e09268cf"} Jul 07 14:07:25.223852 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:25.223815 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" event={"ID":"7e80423b-0747-4914-a00d-19bdf951c3ca","Type":"ContainerStarted","Data":"9b3ec987f4ff964cad7c06b5f7e40549095f8952a248565d556b136abcbf5efe"} Jul 07 14:07:25.225719 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:25.225691 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d4tt5" event={"ID":"d0a6823a-f90b-4bdf-9682-a00e462d88bb","Type":"ContainerStarted","Data":"6e09be7022d05a8ed9ef461581abc289b49165f6559f2b4b826b074e8abbac73"} Jul 07 14:07:25.227039 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:25.227009 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" event={"ID":"2d177a24-96d5-4871-a4c9-e9d031ea97cc","Type":"ContainerStarted","Data":"230f6a94a84b41c59efe363ae78bfb7e88e06e6db255490b39b6115bbd0c992e"} Jul 07 14:07:25.229280 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:25.229237 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9wbnm" event={"ID":"dc2a0c37-b521-42e0-9e34-bcdc75a54636","Type":"ContainerStarted","Data":"a7ab9aec206d424dc54e84a699cff73569ac5a94931507afa998cd98696f50af"} Jul 07 14:07:25.230697 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:25.230663 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" event={"ID":"7e36c76a-c29f-42bc-9e51-4516e19d345a","Type":"ContainerStarted","Data":"ba79e5e9cb25a4233c2a8e882cd07c095e4a1111f0627b6c3b3f1e36d135a80a"} Jul 07 14:07:25.231917 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:25.231894 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6bc6n" event={"ID":"1ac773f2-d330-4140-97aa-8ac19ee4d555","Type":"ContainerStarted","Data":"82208e3a0e35ab626c3f2ac2061cd9ef3c188145d9079f015f33752e049d5090"} Jul 07 14:07:25.233452 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:25.233421 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpdnk" event={"ID":"6cf123ea-2a30-4535-971f-0833cafaedea","Type":"ContainerStarted","Data":"4d0ad0cbeb4b941b3676eb13e22cf72a16bd18b42df612d14ed65e12e581955d"} Jul 07 14:07:25.235143 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:25.235117 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5kt5s" event={"ID":"1053d78f-a5ae-4acd-9a99-32beb279bbb4","Type":"ContainerStarted","Data":"cda1b4da4d097f9f939cf4dad1da0a965cbcbaae18ee4e34b992c574dff5fb7a"} Jul 07 14:07:25.619996 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:25.619954 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:25.620211 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:25.620122 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:07:25.620211 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:25.620191 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret podName:2b457606-cff0-47a1-8938-778f59dd934d nodeName:}" failed. No retries permitted until 2026-07-07 14:07:27.620171278 +0000 UTC m=+6.052413781 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret") pod "global-pull-secret-syncer-xhkww" (UID: "2b457606-cff0-47a1-8938-778f59dd934d") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:07:25.720748 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:25.720707 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs\") pod \"network-metrics-daemon-8rf2w\" (UID: \"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603\") " pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:25.720950 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:25.720790 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtlx2\" (UniqueName: \"kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2\") pod \"network-check-target-vsfxs\" (UID: \"2c345a9e-d35d-40b1-91c5-553c42d60e8f\") " pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:25.721015 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:25.720960 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:07:25.721015 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:25.720986 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:07:25.721015 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:25.721012 2560 projected.go:196] Error preparing data for projected volume kube-api-access-wtlx2 for pod openshift-network-diagnostics/network-check-target-vsfxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:07:25.721178 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:25.721078 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2 podName:2c345a9e-d35d-40b1-91c5-553c42d60e8f nodeName:}" failed. No retries permitted until 2026-07-07 14:07:27.721058009 +0000 UTC m=+6.153300496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wtlx2" (UniqueName: "kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2") pod "network-check-target-vsfxs" (UID: "2c345a9e-d35d-40b1-91c5-553c42d60e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:07:25.721553 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:25.721531 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:07:25.721656 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:25.721600 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs podName:3d5ba7ef-4b44-4c07-9dc0-77a3e4223603 nodeName:}" failed. No retries permitted until 2026-07-07 14:07:27.72157445 +0000 UTC m=+6.153816935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs") pod "network-metrics-daemon-8rf2w" (UID: "3d5ba7ef-4b44-4c07-9dc0-77a3e4223603") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:07:26.032799 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:26.032669 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:02:24 +0000 UTC" deadline="2028-03-10 12:17:34.435612391 +0000 UTC" Jul 07 14:07:26.032799 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:26.032715 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14686h10m8.40290314s" Jul 07 14:07:26.203298 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:26.202991 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:26.203451 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:26.203421 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:26.203545 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:26.203482 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:26.203545 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:26.203509 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:26.203662 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:26.203588 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:26.203734 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:26.203680 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:26.249724 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:26.249557 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-57.ec2.internal" event={"ID":"9386b51d35694e977222ff73c9ef8b76","Type":"ContainerStarted","Data":"fe29e53c69b65a0644520a1cd5bc87ef820d54edf89935c8e4ba9dfaffeefdd0"} Jul 07 14:07:27.257482 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:27.257148 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal" event={"ID":"cf520380696161bcb2a0b72df4da4cb4","Type":"ContainerStarted","Data":"8147aa35105cf46262f8f3dca43d3534b4d46c4b65a6d0a4c1d10b9704f86176"} Jul 07 14:07:27.274869 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:27.273626 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-57.ec2.internal" podStartSLOduration=3.273603275 podStartE2EDuration="3.273603275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:07:26.267519707 +0000 UTC m=+4.699762213" watchObservedRunningTime="2026-07-07 14:07:27.273603275 +0000 UTC m=+5.705845783" Jul 07 14:07:27.641694 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:27.641029 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:27.641694 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:27.641232 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:07:27.641694 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:27.641302 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret podName:2b457606-cff0-47a1-8938-778f59dd934d nodeName:}" failed. No retries permitted until 2026-07-07 14:07:31.641281522 +0000 UTC m=+10.073524004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret") pod "global-pull-secret-syncer-xhkww" (UID: "2b457606-cff0-47a1-8938-778f59dd934d") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:07:27.742044 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:27.741384 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtlx2\" (UniqueName: \"kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2\") pod \"network-check-target-vsfxs\" (UID: \"2c345a9e-d35d-40b1-91c5-553c42d60e8f\") " pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:27.742044 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:27.741470 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs\") pod \"network-metrics-daemon-8rf2w\" (UID: \"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603\") " pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:27.742044 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:27.741609 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:07:27.742044 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:27.741670 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs podName:3d5ba7ef-4b44-4c07-9dc0-77a3e4223603 nodeName:}" failed. No retries permitted until 2026-07-07 14:07:31.741650089 +0000 UTC m=+10.173892593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs") pod "network-metrics-daemon-8rf2w" (UID: "3d5ba7ef-4b44-4c07-9dc0-77a3e4223603") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:07:27.742044 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:27.742014 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:07:27.742044 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:27.742035 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:07:27.742044 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:27.742047 2560 projected.go:196] Error preparing data for projected volume kube-api-access-wtlx2 for pod openshift-network-diagnostics/network-check-target-vsfxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:07:27.742754 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:27.742098 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2 podName:2c345a9e-d35d-40b1-91c5-553c42d60e8f nodeName:}" failed. No retries permitted until 2026-07-07 14:07:31.742082605 +0000 UTC m=+10.174325085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wtlx2" (UniqueName: "kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2") pod "network-check-target-vsfxs" (UID: "2c345a9e-d35d-40b1-91c5-553c42d60e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:07:28.205955 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:28.205917 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:28.206168 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:28.206072 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:28.206865 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:28.206500 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:28.206865 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:28.206603 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:28.206865 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:28.206666 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:28.206865 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:28.206739 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:29.270233 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:29.269345 2560 generic.go:356] "Generic (PLEG): container finished" podID="cf520380696161bcb2a0b72df4da4cb4" containerID="8147aa35105cf46262f8f3dca43d3534b4d46c4b65a6d0a4c1d10b9704f86176" exitCode=0 Jul 07 14:07:29.270233 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:29.269406 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal" event={"ID":"cf520380696161bcb2a0b72df4da4cb4","Type":"ContainerDied","Data":"8147aa35105cf46262f8f3dca43d3534b4d46c4b65a6d0a4c1d10b9704f86176"} Jul 07 14:07:30.204009 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:30.203924 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:30.204009 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:30.203955 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:30.204276 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:30.204079 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:30.204622 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:30.204464 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:30.204622 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:30.203924 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:30.204622 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:30.204589 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:31.686455 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:31.685690 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:31.686455 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:31.685865 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:07:31.686455 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:31.685939 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret podName:2b457606-cff0-47a1-8938-778f59dd934d nodeName:}" failed. No retries permitted until 2026-07-07 14:07:39.685918119 +0000 UTC m=+18.118160603 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret") pod "global-pull-secret-syncer-xhkww" (UID: "2b457606-cff0-47a1-8938-778f59dd934d") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:07:31.786841 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:31.786796 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs\") pod \"network-metrics-daemon-8rf2w\" (UID: \"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603\") " pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:31.787022 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:31.786869 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtlx2\" (UniqueName: \"kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2\") pod \"network-check-target-vsfxs\" (UID: \"2c345a9e-d35d-40b1-91c5-553c42d60e8f\") " pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:31.787022 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:31.787001 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:07:31.787022 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:31.787009 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:07:31.787202 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:31.787024 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:07:31.787202 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:31.787043 2560 projected.go:196] Error preparing data for projected volume kube-api-access-wtlx2 for pod openshift-network-diagnostics/network-check-target-vsfxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:07:31.787202 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:31.787095 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs podName:3d5ba7ef-4b44-4c07-9dc0-77a3e4223603 nodeName:}" failed. No retries permitted until 2026-07-07 14:07:39.787070278 +0000 UTC m=+18.219312775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs") pod "network-metrics-daemon-8rf2w" (UID: "3d5ba7ef-4b44-4c07-9dc0-77a3e4223603") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:07:31.787202 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:31.787120 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2 podName:2c345a9e-d35d-40b1-91c5-553c42d60e8f nodeName:}" failed. No retries permitted until 2026-07-07 14:07:39.787107682 +0000 UTC m=+18.219350173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wtlx2" (UniqueName: "kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2") pod "network-check-target-vsfxs" (UID: "2c345a9e-d35d-40b1-91c5-553c42d60e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:07:32.205094 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:32.204367 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:32.205094 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:32.204471 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:32.205094 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:32.204825 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:32.205094 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:32.204916 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:32.205094 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:32.204982 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:32.205094 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:32.205059 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:34.206332 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:34.206305 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:34.206691 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:34.206399 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:34.206749 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:34.206738 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:34.206819 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:34.206800 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:34.206892 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:34.206859 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:34.206940 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:34.206925 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:36.206679 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:36.206596 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:36.206679 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:36.206632 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:36.207089 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:36.206733 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:36.207089 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:36.206790 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:36.207089 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:36.206894 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:36.207089 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:36.206964 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:38.208249 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:38.207693 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:38.208249 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:38.207837 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:38.208249 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:38.207945 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:38.208808 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:38.208373 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:38.208808 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:38.208407 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:38.208808 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:38.208479 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:39.748529 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:39.748484 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:39.748980 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:39.748645 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:07:39.748980 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:39.748722 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret podName:2b457606-cff0-47a1-8938-778f59dd934d nodeName:}" failed. No retries permitted until 2026-07-07 14:07:55.748702996 +0000 UTC m=+34.180945500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret") pod "global-pull-secret-syncer-xhkww" (UID: "2b457606-cff0-47a1-8938-778f59dd934d") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:07:39.850158 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:39.849330 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs\") pod \"network-metrics-daemon-8rf2w\" (UID: \"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603\") " pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:39.850158 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:39.849402 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtlx2\" (UniqueName: \"kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2\") pod \"network-check-target-vsfxs\" (UID: \"2c345a9e-d35d-40b1-91c5-553c42d60e8f\") " pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:39.850158 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:39.849577 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:07:39.850158 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:39.849598 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:07:39.850158 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:39.849612 2560 projected.go:196] Error preparing data for projected volume kube-api-access-wtlx2 for pod openshift-network-diagnostics/network-check-target-vsfxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:07:39.850158 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:39.849673 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2 podName:2c345a9e-d35d-40b1-91c5-553c42d60e8f nodeName:}" failed. No retries permitted until 2026-07-07 14:07:55.849654946 +0000 UTC m=+34.281897441 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wtlx2" (UniqueName: "kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2") pod "network-check-target-vsfxs" (UID: "2c345a9e-d35d-40b1-91c5-553c42d60e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:07:39.850158 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:39.850065 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:07:39.850158 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:39.850112 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs podName:3d5ba7ef-4b44-4c07-9dc0-77a3e4223603 nodeName:}" failed. No retries permitted until 2026-07-07 14:07:55.850098503 +0000 UTC m=+34.282340986 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs") pod "network-metrics-daemon-8rf2w" (UID: "3d5ba7ef-4b44-4c07-9dc0-77a3e4223603") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:07:40.203666 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:40.203634 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:40.203839 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:40.203645 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:40.203839 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:40.203751 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:40.203839 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:40.203752 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:40.203839 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:40.203834 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:40.204040 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:40.203922 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:42.204330 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:42.204284 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:42.204789 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:42.204429 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:42.207551 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:42.205004 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:42.207551 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:42.205120 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:42.207551 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:42.205242 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:42.207551 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:42.205379 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:44.203156 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:44.203129 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:44.203156 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:44.203158 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:44.203552 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:44.203132 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:44.203552 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:44.203255 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:44.203552 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:44.203333 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:44.203552 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:44.203410 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:45.306113 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.305785 2560 generic.go:356] "Generic (PLEG): container finished" podID="6cf123ea-2a30-4535-971f-0833cafaedea" containerID="04fd037e64355d67386d4e178cadb3ec5a0a7267b1539d9e64ac7e04b5110bca" exitCode=0 Jul 07 14:07:45.306996 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.306161 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpdnk" event={"ID":"6cf123ea-2a30-4535-971f-0833cafaedea","Type":"ContainerDied","Data":"04fd037e64355d67386d4e178cadb3ec5a0a7267b1539d9e64ac7e04b5110bca"} Jul 07 14:07:45.307924 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.307772 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5kt5s" event={"ID":"1053d78f-a5ae-4acd-9a99-32beb279bbb4","Type":"ContainerStarted","Data":"b649338855c6cf3337a6cda03641e2db921ea5f3a5c438a18b05f8937b590061"} Jul 07 14:07:45.309415 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.309393 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vqk55" event={"ID":"79313367-388a-4b7a-9a7c-88cf9aeb1d2a","Type":"ContainerStarted","Data":"43602d15f07eb02b67c94d5adf53244ed0cba2087c6b78b7b3a8dcf6c3db40e2"} Jul 07 14:07:45.311141 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.311119 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" event={"ID":"7e80423b-0747-4914-a00d-19bdf951c3ca","Type":"ContainerStarted","Data":"1dd8159d39339de12ef3dd33dd0b53c4176e7590b88376e649253c663ded9ab1"} Jul 07 14:07:45.312688 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.312666 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" event={"ID":"2d177a24-96d5-4871-a4c9-e9d031ea97cc","Type":"ContainerStarted","Data":"3d7f2c0ed10f439cda14c12f09c9097bdfd55958b1b46dd7767a127c2ac162ce"} Jul 07 14:07:45.314117 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.314093 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9wbnm" event={"ID":"dc2a0c37-b521-42e0-9e34-bcdc75a54636","Type":"ContainerStarted","Data":"ddabb41c926b8d522ae19c56305208b853438f7a997cfffddb0ce5df5c0a4da5"} Jul 07 14:07:45.317450 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.317428 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" event={"ID":"7e36c76a-c29f-42bc-9e51-4516e19d345a","Type":"ContainerStarted","Data":"daa49dd5fe84675dab47d3a73de1e9395f424a6de73e3fac575212977507ba78"} Jul 07 14:07:45.317547 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.317493 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" event={"ID":"7e36c76a-c29f-42bc-9e51-4516e19d345a","Type":"ContainerStarted","Data":"d8af5a314b477f1b433e49975b3da68bd6c23edb5e72fe0c7067cf3b504119d8"} Jul 07 14:07:45.317547 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.317526 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" event={"ID":"7e36c76a-c29f-42bc-9e51-4516e19d345a","Type":"ContainerStarted","Data":"e5c6407ef60f93027f1753a85229b60d2ce74eba9052294fe453bb546d5230af"} Jul 07 14:07:45.317660 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.317565 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" event={"ID":"7e36c76a-c29f-42bc-9e51-4516e19d345a","Type":"ContainerStarted","Data":"47f14b886baa0f2c71435a91427fa4f82cf7dd5e4421eaeb2b2295fd189b848c"} Jul 07 14:07:45.317660 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.317606 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" event={"ID":"7e36c76a-c29f-42bc-9e51-4516e19d345a","Type":"ContainerStarted","Data":"f04291dc2863deebbaf056736af28ec93b3333f36045221702fff15632cb6fa5"} Jul 07 14:07:45.317660 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.317654 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" event={"ID":"7e36c76a-c29f-42bc-9e51-4516e19d345a","Type":"ContainerStarted","Data":"699eb9c83f562bb19c5312484e9adf6a9c1c3678a141f3d7450102935aee1630"} Jul 07 14:07:45.318986 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.318964 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6bc6n" event={"ID":"1ac773f2-d330-4140-97aa-8ac19ee4d555","Type":"ContainerStarted","Data":"9cfbbeffe2639f815e79843c0c3b1f71642139e8452ce4b4b6f60922e362d1d9"} Jul 07 14:07:45.320851 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.320832 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal" event={"ID":"cf520380696161bcb2a0b72df4da4cb4","Type":"ContainerStarted","Data":"05aa797918f8e95ba99b5eb3c57c5e597cfe65e3e16efefe3bc72dec62b24b68"} Jul 07 14:07:45.340722 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.340679 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6bc6n" podStartSLOduration=2.231653378 podStartE2EDuration="22.340667533s" totalImagesPullingTime="20.109014155s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:07:45.340656584 +0000 UTC m=+23.772899087" watchObservedRunningTime="2026-07-07 14:07:45.340667533 +0000 UTC m=+23.772910034" Jul 07 14:07:45.357569 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.357527 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vqk55" podStartSLOduration=2.216794973 podStartE2EDuration="22.357514443s" totalImagesPullingTime="20.14071947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:07:45.357320004 +0000 UTC m=+23.789562507" watchObservedRunningTime="2026-07-07 14:07:45.357514443 +0000 UTC m=+23.789756946" Jul 07 14:07:45.370598 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.370553 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5kt5s" podStartSLOduration=6.832459527 podStartE2EDuration="22.370541551s" totalImagesPullingTime="15.538082024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:07:45.369896701 +0000 UTC m=+23.802139201" watchObservedRunningTime="2026-07-07 14:07:45.370541551 +0000 UTC m=+23.802784053" Jul 07 14:07:45.392018 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.391980 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qjlhn" podStartSLOduration=2.271895546 podStartE2EDuration="22.391969067s" totalImagesPullingTime="20.120073521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:07:45.391532626 +0000 UTC m=+23.823775126" watchObservedRunningTime="2026-07-07 14:07:45.391969067 +0000 UTC m=+23.824211568" Jul 07 14:07:45.407167 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.407132 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9wbnm" podStartSLOduration=10.901241221 podStartE2EDuration="22.407121403s" totalImagesPullingTime="11.505880182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:07:45.406798001 +0000 UTC m=+23.839040502" watchObservedRunningTime="2026-07-07 14:07:45.407121403 +0000 UTC m=+23.839363905" Jul 07 14:07:45.423593 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.421781 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-57.ec2.internal" podStartSLOduration=21.421765958 podStartE2EDuration="21.421765958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:07:45.421394162 +0000 UTC m=+23.853636664" watchObservedRunningTime="2026-07-07 14:07:45.421765958 +0000 UTC m=+23.854008462" Jul 07 14:07:45.495141 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.495112 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:07:45.580474 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.580381 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:07:45.778422 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:45.778400 2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 14:07:46.065433 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:46.065328 2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T14:07:45.778417412Z","UUID":"0179ea54-a58a-4b11-96aa-9fcb47c0ef5b","Handler":null,"Name":"","Endpoint":""} Jul 07 14:07:46.067238 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:46.067211 2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 14:07:46.067389 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:46.067258 2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 14:07:46.202973 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:46.202940 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:46.202973 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:46.202985 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:46.203214 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:46.202940 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:46.203214 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:46.203079 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:46.203214 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:46.203128 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:46.203363 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:46.203228 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:46.324381 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:46.324342 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d4tt5" event={"ID":"d0a6823a-f90b-4bdf-9682-a00e462d88bb","Type":"ContainerStarted","Data":"2d19bac69f91f1f39f62f993a93980a623e09aca920612fe2e27182ffda86830"} Jul 07 14:07:46.326511 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:46.326485 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" event={"ID":"2d177a24-96d5-4871-a4c9-e9d031ea97cc","Type":"ContainerStarted","Data":"c557ccc31f1421a3744dd56b2d60d96533062292b3129c735f0c4483eb96f871"} Jul 07 14:07:46.338816 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:46.338775 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d4tt5" podStartSLOduration=3.210359498 podStartE2EDuration="23.338758977s" totalImagesPullingTime="20.128399479s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:07:46.338394568 +0000 UTC m=+24.770637071" watchObservedRunningTime="2026-07-07 14:07:46.338758977 +0000 UTC m=+24.771001480" Jul 07 14:07:46.485378 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:46.485334 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:07:46.547320 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:46.547280 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:07:47.331414 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:47.331374 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" event={"ID":"2d177a24-96d5-4871-a4c9-e9d031ea97cc","Type":"ContainerStarted","Data":"03141c0987ca6a048bfe378773e852995b3ae0bc8e6717e87136cabc8571f389"} Jul 07 14:07:47.335076 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:47.335043 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" event={"ID":"7e36c76a-c29f-42bc-9e51-4516e19d345a","Type":"ContainerStarted","Data":"750a6cecd7f86789b6991a3f1947dea406e98c8ec026531fdec45b8ed20c2d23"} Jul 07 14:07:47.349582 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:47.349522 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kpgck" podStartSLOduration=2.364742427 podStartE2EDuration="24.349503077s" totalImagesPullingTime="21.98476065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:07:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:07:47.349237797 +0000 UTC m=+25.781480300" watchObservedRunningTime="2026-07-07 14:07:47.349503077 +0000 UTC m=+25.781745580" Jul 07 14:07:47.847434 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:47.847398 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9wbnm" Jul 07 14:07:47.848241 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:47.848219 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9wbnm" Jul 07 14:07:48.202802 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:48.202729 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:48.202928 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:48.202729 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:48.202928 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:48.202837 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:48.203014 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:48.202945 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:48.203014 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:48.202738 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:48.203084 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:48.203045 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:49.343991 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:49.343730 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" event={"ID":"7e36c76a-c29f-42bc-9e51-4516e19d345a","Type":"ContainerStarted","Data":"3b5d2400a600d2d37aa0b16b5f59d05f190474a690258ea88ce8ece9ee2872f0"} Jul 07 14:07:49.344661 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:49.344073 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:49.361628 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:49.361609 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:49.373074 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:49.373031 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" podStartSLOduration=6.192023825 podStartE2EDuration="26.373017117s" totalImagesPullingTime="20.180993292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:07:49.372527559 +0000 UTC m=+27.804770060" watchObservedRunningTime="2026-07-07 14:07:49.373017117 +0000 UTC m=+27.805259619" Jul 07 14:07:50.186286 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:50.186233 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:07:50.203138 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:50.203111 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:50.203138 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:50.203129 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:50.203345 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:50.203112 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:50.203345 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:50.203231 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:50.203345 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:50.203313 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:50.203496 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:50.203384 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:50.285999 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:50.285956 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:07:50.346282 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:50.346235 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:50.346944 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:50.346325 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:50.362484 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:50.362459 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:07:51.228086 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:51.227840 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vsfxs"] Jul 07 14:07:51.228316 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:51.228187 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:51.228394 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:51.228371 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:51.230559 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:51.230535 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8rf2w"] Jul 07 14:07:51.230671 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:51.230635 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:51.230760 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:51.230740 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:51.239668 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:51.239641 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xhkww"] Jul 07 14:07:51.239783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:51.239730 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:51.239852 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:51.239834 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:52.353417 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:52.353378 2560 generic.go:356] "Generic (PLEG): container finished" podID="6cf123ea-2a30-4535-971f-0833cafaedea" containerID="b4d12a22806dbd6739e44cb85ee5e35f318264f5199673d1bca2775f1a2d8be3" exitCode=0 Jul 07 14:07:52.353935 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:52.353896 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpdnk" event={"ID":"6cf123ea-2a30-4535-971f-0833cafaedea","Type":"ContainerDied","Data":"b4d12a22806dbd6739e44cb85ee5e35f318264f5199673d1bca2775f1a2d8be3"} Jul 07 14:07:53.177817 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:53.177784 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9wbnm" Jul 07 14:07:53.177985 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:53.177907 2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 14:07:53.178320 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:53.178306 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9wbnm" Jul 07 14:07:53.203034 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:53.203007 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:53.203134 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:53.203092 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:53.203134 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:53.203097 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:53.203134 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:53.203129 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:53.203310 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:53.203276 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:53.203440 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:53.203376 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:53.243174 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:53.243137 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:07:53.452988 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:53.452904 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:07:54.023255 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:54.023215 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:07:54.360472 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:54.360256 2560 generic.go:356] "Generic (PLEG): container finished" podID="6cf123ea-2a30-4535-971f-0833cafaedea" containerID="3163c238e46f393d678c8c0c38f82ac06353a66833445d9925effb22a4363a2d" exitCode=0 Jul 07 14:07:54.360584 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:54.360362 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpdnk" event={"ID":"6cf123ea-2a30-4535-971f-0833cafaedea","Type":"ContainerDied","Data":"3163c238e46f393d678c8c0c38f82ac06353a66833445d9925effb22a4363a2d"} Jul 07 14:07:54.574158 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:54.574123 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:07:54.635321 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:54.635290 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:07:55.203044 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:55.203013 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:55.203233 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:55.203057 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:55.203233 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:55.203057 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:55.203233 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:55.203136 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:55.203419 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:55.203227 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:55.203419 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:55.203384 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:55.764330 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:55.764299 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:55.764704 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:55.764422 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:07:55.764704 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:55.764463 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret podName:2b457606-cff0-47a1-8938-778f59dd934d nodeName:}" failed. No retries permitted until 2026-07-07 14:08:27.764450984 +0000 UTC m=+66.196693469 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret") pod "global-pull-secret-syncer-xhkww" (UID: "2b457606-cff0-47a1-8938-778f59dd934d") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:07:55.864890 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:55.864863 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs\") pod \"network-metrics-daemon-8rf2w\" (UID: \"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603\") " pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:55.865015 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:55.864907 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtlx2\" (UniqueName: \"kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2\") pod \"network-check-target-vsfxs\" (UID: \"2c345a9e-d35d-40b1-91c5-553c42d60e8f\") " pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:55.865061 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:55.865041 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:07:55.865061 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:55.865040 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:07:55.865156 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:55.865058 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:07:55.865156 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:55.865073 2560 projected.go:196] Error preparing data for projected volume kube-api-access-wtlx2 for pod openshift-network-diagnostics/network-check-target-vsfxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:07:55.865156 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:55.865121 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs podName:3d5ba7ef-4b44-4c07-9dc0-77a3e4223603 nodeName:}" failed. No retries permitted until 2026-07-07 14:08:27.865104467 +0000 UTC m=+66.297346952 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs") pod "network-metrics-daemon-8rf2w" (UID: "3d5ba7ef-4b44-4c07-9dc0-77a3e4223603") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:07:55.865156 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:55.865137 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2 podName:2c345a9e-d35d-40b1-91c5-553c42d60e8f nodeName:}" failed. No retries permitted until 2026-07-07 14:08:27.865130776 +0000 UTC m=+66.297373259 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wtlx2" (UniqueName: "kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2") pod "network-check-target-vsfxs" (UID: "2c345a9e-d35d-40b1-91c5-553c42d60e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:07:56.367702 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:56.367668 2560 generic.go:356] "Generic (PLEG): container finished" podID="6cf123ea-2a30-4535-971f-0833cafaedea" containerID="24b779c37d1dd535fb04f782cb4da0dff2d164096e12265915825491465063fb" exitCode=0 Jul 07 14:07:56.367839 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:56.367734 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpdnk" event={"ID":"6cf123ea-2a30-4535-971f-0833cafaedea","Type":"ContainerDied","Data":"24b779c37d1dd535fb04f782cb4da0dff2d164096e12265915825491465063fb"} Jul 07 14:07:57.203179 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.203147 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:57.203631 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.203248 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:57.203631 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:57.203251 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vsfxs" podUID="2c345a9e-d35d-40b1-91c5-553c42d60e8f" Jul 07 14:07:57.203631 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.203283 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:57.203631 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:57.203386 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhkww" podUID="2b457606-cff0-47a1-8938-778f59dd934d" Jul 07 14:07:57.203631 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:57.203493 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:07:57.387998 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.387915 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-57.ec2.internal" event="NodeReady" Jul 07 14:07:57.388149 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.388066 2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 14:07:57.437010 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.436977 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5cgb6"] Jul 07 14:07:57.466427 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.466397 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hkczl"] Jul 07 14:07:57.466573 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.466520 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5cgb6" Jul 07 14:07:57.469128 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.469100 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sfsdl\"" Jul 07 14:07:57.472342 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.470046 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 14:07:57.472342 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.470238 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 14:07:57.472342 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.470837 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 14:07:57.492603 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.492579 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5cgb6"] Jul 07 14:07:57.492722 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.492617 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hkczl"] Jul 07 14:07:57.492722 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.492676 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hkczl" Jul 07 14:07:57.495100 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.495081 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 14:07:57.495100 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.495095 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cxvss\"" Jul 07 14:07:57.495259 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.495084 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 14:07:57.559154 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.559113 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:07:57.581073 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.581040 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b88eafd6-43c8-460b-b450-7b2714fb1a9c-config-volume\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:07:57.581283 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.581111 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:07:57.581283 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.581137 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88eafd6-43c8-460b-b450-7b2714fb1a9c-tmp-dir\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:07:57.581283 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.581175 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a334bf52-3bf3-4862-8680-d4e89f3d28bc-cert\") pod \"ingress-canary-5cgb6\" (UID: \"a334bf52-3bf3-4862-8680-d4e89f3d28bc\") " pod="openshift-ingress-canary/ingress-canary-5cgb6" Jul 07 14:07:57.581283 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.581201 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dsg9\" (UniqueName: \"kubernetes.io/projected/a334bf52-3bf3-4862-8680-d4e89f3d28bc-kube-api-access-5dsg9\") pod \"ingress-canary-5cgb6\" (UID: \"a334bf52-3bf3-4862-8680-d4e89f3d28bc\") " pod="openshift-ingress-canary/ingress-canary-5cgb6" Jul 07 14:07:57.581283 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.581248 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzchd\" (UniqueName: \"kubernetes.io/projected/b88eafd6-43c8-460b-b450-7b2714fb1a9c-kube-api-access-tzchd\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:07:57.616091 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.616040 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:07:57.682209 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.682123 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:07:57.682209 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.682166 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88eafd6-43c8-460b-b450-7b2714fb1a9c-tmp-dir\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:07:57.682209 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.682200 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a334bf52-3bf3-4862-8680-d4e89f3d28bc-cert\") pod \"ingress-canary-5cgb6\" (UID: \"a334bf52-3bf3-4862-8680-d4e89f3d28bc\") " pod="openshift-ingress-canary/ingress-canary-5cgb6" Jul 07 14:07:57.682472 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.682228 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dsg9\" (UniqueName: \"kubernetes.io/projected/a334bf52-3bf3-4862-8680-d4e89f3d28bc-kube-api-access-5dsg9\") pod \"ingress-canary-5cgb6\" (UID: \"a334bf52-3bf3-4862-8680-d4e89f3d28bc\") " pod="openshift-ingress-canary/ingress-canary-5cgb6" Jul 07 14:07:57.682472 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:57.682254 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:07:57.682472 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.682296 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzchd\" (UniqueName: \"kubernetes.io/projected/b88eafd6-43c8-460b-b450-7b2714fb1a9c-kube-api-access-tzchd\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:07:57.682472 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:57.682345 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls podName:b88eafd6-43c8-460b-b450-7b2714fb1a9c nodeName:}" failed. No retries permitted until 2026-07-07 14:07:58.182325528 +0000 UTC m=+36.614568019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls") pod "dns-default-hkczl" (UID: "b88eafd6-43c8-460b-b450-7b2714fb1a9c") : secret "dns-default-metrics-tls" not found Jul 07 14:07:57.682472 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.682366 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b88eafd6-43c8-460b-b450-7b2714fb1a9c-config-volume\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:07:57.682666 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.682599 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88eafd6-43c8-460b-b450-7b2714fb1a9c-tmp-dir\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:07:57.682922 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.682896 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b88eafd6-43c8-460b-b450-7b2714fb1a9c-config-volume\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:07:57.687000 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.686976 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a334bf52-3bf3-4862-8680-d4e89f3d28bc-cert\") pod \"ingress-canary-5cgb6\" (UID: \"a334bf52-3bf3-4862-8680-d4e89f3d28bc\") " pod="openshift-ingress-canary/ingress-canary-5cgb6" Jul 07 14:07:57.691094 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.691066 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzchd\" (UniqueName: \"kubernetes.io/projected/b88eafd6-43c8-460b-b450-7b2714fb1a9c-kube-api-access-tzchd\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:07:57.691235 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.691140 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dsg9\" (UniqueName: \"kubernetes.io/projected/a334bf52-3bf3-4862-8680-d4e89f3d28bc-kube-api-access-5dsg9\") pod \"ingress-canary-5cgb6\" (UID: \"a334bf52-3bf3-4862-8680-d4e89f3d28bc\") " pod="openshift-ingress-canary/ingress-canary-5cgb6" Jul 07 14:07:57.778279 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:57.778235 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5cgb6" Jul 07 14:07:58.022068 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:58.022041 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5cgb6"] Jul 07 14:07:58.026082 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:07:58.026056 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda334bf52_3bf3_4862_8680_d4e89f3d28bc.slice/crio-f5de4e4b59e08b0d34b7260bd7af55431841e5b330f7a5fd83917f29a89744ec WatchSource:0}: Error finding container f5de4e4b59e08b0d34b7260bd7af55431841e5b330f7a5fd83917f29a89744ec: Status 404 returned error can't find the container with id f5de4e4b59e08b0d34b7260bd7af55431841e5b330f7a5fd83917f29a89744ec Jul 07 14:07:58.186298 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:58.186245 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:07:58.186474 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:58.186380 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:07:58.186474 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:58.186456 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls podName:b88eafd6-43c8-460b-b450-7b2714fb1a9c nodeName:}" failed. No retries permitted until 2026-07-07 14:07:59.186437301 +0000 UTC m=+37.618679784 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls") pod "dns-default-hkczl" (UID: "b88eafd6-43c8-460b-b450-7b2714fb1a9c") : secret "dns-default-metrics-tls" not found Jul 07 14:07:58.373425 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:58.373395 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5cgb6" event={"ID":"a334bf52-3bf3-4862-8680-d4e89f3d28bc","Type":"ContainerStarted","Data":"f5de4e4b59e08b0d34b7260bd7af55431841e5b330f7a5fd83917f29a89744ec"} Jul 07 14:07:59.196706 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:59.196668 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:07:59.196884 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:59.196869 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:07:59.196943 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:07:59.196935 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls podName:b88eafd6-43c8-460b-b450-7b2714fb1a9c nodeName:}" failed. No retries permitted until 2026-07-07 14:08:01.19691534 +0000 UTC m=+39.629157839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls") pod "dns-default-hkczl" (UID: "b88eafd6-43c8-460b-b450-7b2714fb1a9c") : secret "dns-default-metrics-tls" not found Jul 07 14:07:59.202749 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:59.202719 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:07:59.202878 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:59.202753 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:07:59.203121 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:59.203105 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:07:59.205763 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:59.205344 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 14:07:59.205763 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:59.205648 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wgdzp\"" Jul 07 14:07:59.205884 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:59.205869 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 14:07:59.206042 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:59.206015 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 14:07:59.206282 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:59.206240 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 14:07:59.206282 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:07:59.206281 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d5l84\"" Jul 07 14:08:00.063942 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:00.063907 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:08:00.124362 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:00.124327 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:08:01.213446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:01.213405 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:08:01.213909 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:08:01.213569 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:08:01.213909 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:08:01.213642 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls podName:b88eafd6-43c8-460b-b450-7b2714fb1a9c nodeName:}" failed. No retries permitted until 2026-07-07 14:08:05.213621028 +0000 UTC m=+43.645863520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls") pod "dns-default-hkczl" (UID: "b88eafd6-43c8-460b-b450-7b2714fb1a9c") : secret "dns-default-metrics-tls" not found Jul 07 14:08:02.687019 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:02.686969 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:08:03.388691 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:03.388656 2560 generic.go:356] "Generic (PLEG): container finished" podID="6cf123ea-2a30-4535-971f-0833cafaedea" containerID="b705ed89b1c532e23294133c13d1af4173d39087868a37305b2c319a5453e942" exitCode=0 Jul 07 14:08:03.388844 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:03.388732 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpdnk" event={"ID":"6cf123ea-2a30-4535-971f-0833cafaedea","Type":"ContainerDied","Data":"b705ed89b1c532e23294133c13d1af4173d39087868a37305b2c319a5453e942"} Jul 07 14:08:03.390141 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:03.390116 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5cgb6" event={"ID":"a334bf52-3bf3-4862-8680-d4e89f3d28bc","Type":"ContainerStarted","Data":"4dc338261e3ca634baa1724175d649a11c63e7c0a734a2e9e525c82b9a5ad885"} Jul 07 14:08:03.425971 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:03.425927 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5cgb6" podStartSLOduration=1.9743900380000001 podStartE2EDuration="6.425913264s" totalImagesPullingTime="4.451523226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:07:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:08:03.425487218 +0000 UTC m=+41.857729723" watchObservedRunningTime="2026-07-07 14:08:03.425913264 +0000 UTC m=+41.858155765" Jul 07 14:08:04.395177 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:04.395142 2560 generic.go:356] "Generic (PLEG): container finished" podID="6cf123ea-2a30-4535-971f-0833cafaedea" containerID="d8cefa6800e5204895e175b9db6c9146db52932591fbc7a2eaf9d1315c45bed6" exitCode=0 Jul 07 14:08:04.395794 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:04.395233 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpdnk" event={"ID":"6cf123ea-2a30-4535-971f-0833cafaedea","Type":"ContainerDied","Data":"d8cefa6800e5204895e175b9db6c9146db52932591fbc7a2eaf9d1315c45bed6"} Jul 07 14:08:04.994388 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:04.994351 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:08:05.244670 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:05.244570 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:08:05.244808 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:08:05.244727 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:08:05.244808 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:08:05.244798 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls podName:b88eafd6-43c8-460b-b450-7b2714fb1a9c nodeName:}" failed. No retries permitted until 2026-07-07 14:08:13.244780137 +0000 UTC m=+51.677022622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls") pod "dns-default-hkczl" (UID: "b88eafd6-43c8-460b-b450-7b2714fb1a9c") : secret "dns-default-metrics-tls" not found Jul 07 14:08:05.400977 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:05.400947 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bpdnk" event={"ID":"6cf123ea-2a30-4535-971f-0833cafaedea","Type":"ContainerStarted","Data":"6cf2c037e5b996d12c80e2c73b479cc19ba53acfaa8ca0d9782d053a61fb4878"} Jul 07 14:08:05.426819 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:05.426775 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bpdnk" podStartSLOduration=6.377845596 podStartE2EDuration="42.426755433s" totalImagesPullingTime="36.048909837s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:07:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:08:05.425486547 +0000 UTC m=+43.857729051" watchObservedRunningTime="2026-07-07 14:08:05.426755433 +0000 UTC m=+43.858997936" Jul 07 14:08:05.547237 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:05.547202 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:08:05.647674 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:05.647640 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:08:13.297099 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:13.297055 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:08:13.297572 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:08:13.297209 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:08:13.297572 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:08:13.297287 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls podName:b88eafd6-43c8-460b-b450-7b2714fb1a9c nodeName:}" failed. No retries permitted until 2026-07-07 14:08:29.29725209 +0000 UTC m=+67.729494570 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls") pod "dns-default-hkczl" (UID: "b88eafd6-43c8-460b-b450-7b2714fb1a9c") : secret "dns-default-metrics-tls" not found Jul 07 14:08:16.037430 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:16.037391 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:08:16.093761 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:16.093716 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:08:22.364765 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:22.364738 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wprk2" Jul 07 14:08:22.482236 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:22.482201 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:08:22.543344 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:22.543305 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:08:27.795594 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:27.795548 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:08:27.798405 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:27.798388 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 14:08:27.809792 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:27.809772 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b457606-cff0-47a1-8938-778f59dd934d-original-pull-secret\") pod \"global-pull-secret-syncer-xhkww\" (UID: \"2b457606-cff0-47a1-8938-778f59dd934d\") " pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:08:27.896873 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:27.896828 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtlx2\" (UniqueName: \"kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2\") pod \"network-check-target-vsfxs\" (UID: \"2c345a9e-d35d-40b1-91c5-553c42d60e8f\") " pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:08:27.897035 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:27.896897 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs\") pod \"network-metrics-daemon-8rf2w\" (UID: \"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603\") " pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:08:27.899576 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:27.899556 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 14:08:27.899626 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:27.899558 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 14:08:27.907734 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:08:27.907712 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 14:08:27.907803 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:08:27.907777 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs podName:3d5ba7ef-4b44-4c07-9dc0-77a3e4223603 nodeName:}" failed. No retries permitted until 2026-07-07 14:09:31.907759876 +0000 UTC m=+130.340002359 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs") pod "network-metrics-daemon-8rf2w" (UID: "3d5ba7ef-4b44-4c07-9dc0-77a3e4223603") : secret "metrics-daemon-secret" not found Jul 07 14:08:27.909764 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:27.909749 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 14:08:27.921009 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:27.920981 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtlx2\" (UniqueName: \"kubernetes.io/projected/2c345a9e-d35d-40b1-91c5-553c42d60e8f-kube-api-access-wtlx2\") pod \"network-check-target-vsfxs\" (UID: \"2c345a9e-d35d-40b1-91c5-553c42d60e8f\") " pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:08:28.018370 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:28.018333 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhkww" Jul 07 14:08:28.034643 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:28.034622 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d5l84\"" Jul 07 14:08:28.042336 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:28.042315 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:08:28.153466 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:28.153254 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xhkww"] Jul 07 14:08:28.174561 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:28.174539 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vsfxs"] Jul 07 14:08:28.177487 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:08:28.177463 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c345a9e_d35d_40b1_91c5_553c42d60e8f.slice/crio-7c15653a10a4acd6a3d5d4d620903e00030b380cd4c0c1f5dc193ed895859a18 WatchSource:0}: Error finding container 7c15653a10a4acd6a3d5d4d620903e00030b380cd4c0c1f5dc193ed895859a18: Status 404 returned error can't find the container with id 7c15653a10a4acd6a3d5d4d620903e00030b380cd4c0c1f5dc193ed895859a18 Jul 07 14:08:28.460076 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:28.459987 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vsfxs" event={"ID":"2c345a9e-d35d-40b1-91c5-553c42d60e8f","Type":"ContainerStarted","Data":"7c15653a10a4acd6a3d5d4d620903e00030b380cd4c0c1f5dc193ed895859a18"} Jul 07 14:08:28.461036 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:28.461013 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xhkww" event={"ID":"2b457606-cff0-47a1-8938-778f59dd934d","Type":"ContainerStarted","Data":"44eacce61b1386bde93e3d509613842f027c120555da58ea1cc33c77a8d905b5"} Jul 07 14:08:29.306383 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:29.306346 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:08:29.306951 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:08:29.306518 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:08:29.306951 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:08:29.306579 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls podName:b88eafd6-43c8-460b-b450-7b2714fb1a9c nodeName:}" failed. No retries permitted until 2026-07-07 14:09:01.306558482 +0000 UTC m=+99.738800962 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls") pod "dns-default-hkczl" (UID: "b88eafd6-43c8-460b-b450-7b2714fb1a9c") : secret "dns-default-metrics-tls" not found Jul 07 14:08:33.476590 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:33.476553 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xhkww" event={"ID":"2b457606-cff0-47a1-8938-778f59dd934d","Type":"ContainerStarted","Data":"eab9e2277520d6591f7c41845c0f8c5165ccddc3c90ba52d1347618227ceccac"} Jul 07 14:08:33.478087 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:33.478067 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vsfxs" event={"ID":"2c345a9e-d35d-40b1-91c5-553c42d60e8f","Type":"ContainerStarted","Data":"33db423e8522add973a8568aa14fab75ae1b3cad07c833f3d2459a808e08a228"} Jul 07 14:08:33.478188 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:33.478147 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:08:33.493661 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:33.493611 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xhkww" podStartSLOduration=66.056877259 podStartE2EDuration="1m10.493596485s" totalImagesPullingTime="4.436719226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:08:33.493130215 +0000 UTC m=+71.925372718" watchObservedRunningTime="2026-07-07 14:08:33.493596485 +0000 UTC m=+71.925838988" Jul 07 14:08:33.508512 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:08:33.508466 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vsfxs" podStartSLOduration=67.098481398 podStartE2EDuration="1m11.508445893s" totalImagesPullingTime="4.409964495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:08:33.508128382 +0000 UTC m=+71.940370884" watchObservedRunningTime="2026-07-07 14:08:33.508445893 +0000 UTC m=+71.940688399" Jul 07 14:09:01.344315 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:09:01.344251 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:09:01.344751 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:09:01.344375 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:09:01.344751 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:09:01.344445 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls podName:b88eafd6-43c8-460b-b450-7b2714fb1a9c nodeName:}" failed. No retries permitted until 2026-07-07 14:10:05.344429186 +0000 UTC m=+163.776671670 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls") pod "dns-default-hkczl" (UID: "b88eafd6-43c8-460b-b450-7b2714fb1a9c") : secret "dns-default-metrics-tls" not found Jul 07 14:09:04.482692 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:09:04.482658 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vsfxs" Jul 07 14:09:31.946704 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:09:31.946664 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs\") pod \"network-metrics-daemon-8rf2w\" (UID: \"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603\") " pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:09:31.947157 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:09:31.946785 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 14:09:31.947157 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:09:31.946843 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs podName:3d5ba7ef-4b44-4c07-9dc0-77a3e4223603 nodeName:}" failed. No retries permitted until 2026-07-07 14:11:33.94682938 +0000 UTC m=+252.379071860 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs") pod "network-metrics-daemon-8rf2w" (UID: "3d5ba7ef-4b44-4c07-9dc0-77a3e4223603") : secret "metrics-daemon-secret" not found Jul 07 14:09:57.053033 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:09:57.052986 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:00.502017 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:00.501982 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hkczl" podUID="b88eafd6-43c8-460b-b450-7b2714fb1a9c" Jul 07 14:10:00.681209 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:00.681178 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hkczl" Jul 07 14:10:02.225866 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:02.225823 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8rf2w" podUID="3d5ba7ef-4b44-4c07-9dc0-77a3e4223603" Jul 07 14:10:05.356494 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:05.356457 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:10:05.356866 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:05.356575 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 14:10:05.356866 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:05.356637 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls podName:b88eafd6-43c8-460b-b450-7b2714fb1a9c nodeName:}" failed. No retries permitted until 2026-07-07 14:12:07.356623533 +0000 UTC m=+285.788866022 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls") pod "dns-default-hkczl" (UID: "b88eafd6-43c8-460b-b450-7b2714fb1a9c") : secret "dns-default-metrics-tls" not found Jul 07 14:10:13.350904 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:13.350832 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:17.203453 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:17.203398 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:10:23.966699 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.966657 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5"] Jul 07 14:10:23.968759 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.968741 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:23.969986 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.969964 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq"] Jul 07 14:10:23.971847 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.971827 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" Jul 07 14:10:23.972254 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.972234 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 14:10:23.972375 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.972348 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 07 14:10:23.973009 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.972987 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh"] Jul 07 14:10:23.973095 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.973011 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jlvzf\"" Jul 07 14:10:23.973586 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.973567 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 14:10:23.974958 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.974941 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" Jul 07 14:10:23.975140 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.975117 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 07 14:10:23.975140 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.975130 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:10:23.975335 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.975318 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 07 14:10:23.975590 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.975576 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 07 14:10:23.975752 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.975734 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vfck4\"" Jul 07 14:10:23.975816 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.975783 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 07 14:10:23.978027 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.978004 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:10:23.978116 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.978071 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 07 14:10:23.978385 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.978373 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cmbht\"" Jul 07 14:10:23.979771 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.979755 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 07 14:10:23.989735 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.989715 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq"] Jul 07 14:10:23.992428 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.992406 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5"] Jul 07 14:10:23.999207 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:23.999189 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh"] Jul 07 14:10:24.066145 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.066115 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-hwj87"] Jul 07 14:10:24.068052 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.068036 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.070294 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.070252 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68c89555f7-pxm8l"] Jul 07 14:10:24.072035 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.072019 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.072953 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.072938 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 07 14:10:24.073242 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.073223 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 14:10:24.073334 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.073321 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-426vz\"" Jul 07 14:10:24.073564 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.073545 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 14:10:24.073835 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.073808 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 07 14:10:24.077101 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.077074 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 14:10:24.077403 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.077383 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xtlr6\"" Jul 07 14:10:24.077780 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.077752 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 14:10:24.078615 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.078593 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-w76w5\" (UID: \"fd5fab7c-5de3-4f92-85a9-863fd04af85d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:24.078763 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.078746 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c520cb66-9856-40eb-981c-ad7a9687928a-config\") pod \"service-ca-operator-5d46d9ffd4-nrjbq\" (UID: \"c520cb66-9856-40eb-981c-ad7a9687928a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" Jul 07 14:10:24.078883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.078865 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c520cb66-9856-40eb-981c-ad7a9687928a-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-nrjbq\" (UID: \"c520cb66-9856-40eb-981c-ad7a9687928a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" Jul 07 14:10:24.078989 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.078974 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpq2p\" (UniqueName: \"kubernetes.io/projected/fd5fab7c-5de3-4f92-85a9-863fd04af85d-kube-api-access-hpq2p\") pod \"cluster-monitoring-operator-64786d9cbb-w76w5\" (UID: \"fd5fab7c-5de3-4f92-85a9-863fd04af85d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:24.079135 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.079104 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-dhvrh\" (UID: \"1dd48565-d6eb-406e-beb0-4eb814d1b0f5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" Jul 07 14:10:24.079254 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.079153 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvnq2\" (UniqueName: \"kubernetes.io/projected/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-kube-api-access-mvnq2\") pod \"cluster-samples-operator-56fcf86bf-dhvrh\" (UID: \"1dd48565-d6eb-406e-beb0-4eb814d1b0f5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" Jul 07 14:10:24.079461 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.079426 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n54bw\" (UniqueName: \"kubernetes.io/projected/c520cb66-9856-40eb-981c-ad7a9687928a-kube-api-access-n54bw\") pod \"service-ca-operator-5d46d9ffd4-nrjbq\" (UID: \"c520cb66-9856-40eb-981c-ad7a9687928a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" Jul 07 14:10:24.079661 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.079644 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd5fab7c-5de3-4f92-85a9-863fd04af85d-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-w76w5\" (UID: \"fd5fab7c-5de3-4f92-85a9-863fd04af85d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:24.081085 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.081065 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 14:10:24.085664 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.085631 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 14:10:24.086214 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.086191 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 07 14:10:24.089309 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.089290 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-hwj87"] Jul 07 14:10:24.108629 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.108599 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68c89555f7-pxm8l"] Jul 07 14:10:24.180636 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.180599 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-tmp\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.180812 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.180656 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-serving-cert\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.180812 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.180676 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-snapshots\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.180812 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.180692 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.180812 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.180722 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.180812 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.180768 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd5fab7c-5de3-4f92-85a9-863fd04af85d-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-w76w5\" (UID: \"fd5fab7c-5de3-4f92-85a9-863fd04af85d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:24.180812 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.180804 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c520cb66-9856-40eb-981c-ad7a9687928a-config\") pod \"service-ca-operator-5d46d9ffd4-nrjbq\" (UID: \"c520cb66-9856-40eb-981c-ad7a9687928a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" Jul 07 14:10:24.181071 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.180827 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.181071 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.180845 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgxkn\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-kube-api-access-pgxkn\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.181071 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.180863 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c520cb66-9856-40eb-981c-ad7a9687928a-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-nrjbq\" (UID: \"c520cb66-9856-40eb-981c-ad7a9687928a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" Jul 07 14:10:24.181071 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.180880 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpq2p\" (UniqueName: \"kubernetes.io/projected/fd5fab7c-5de3-4f92-85a9-863fd04af85d-kube-api-access-hpq2p\") pod \"cluster-monitoring-operator-64786d9cbb-w76w5\" (UID: \"fd5fab7c-5de3-4f92-85a9-863fd04af85d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:24.181071 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.180896 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98855958-0a9b-47ca-93a9-8b82df6c7ee6-ca-trust-extracted\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.181071 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.180927 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgvbv\" (UniqueName: \"kubernetes.io/projected/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-kube-api-access-bgvbv\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.181071 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.181035 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-bound-sa-token\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.181445 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.181100 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-dhvrh\" (UID: \"1dd48565-d6eb-406e-beb0-4eb814d1b0f5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" Jul 07 14:10:24.181445 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.181129 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnq2\" (UniqueName: \"kubernetes.io/projected/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-kube-api-access-mvnq2\") pod \"cluster-samples-operator-56fcf86bf-dhvrh\" (UID: \"1dd48565-d6eb-406e-beb0-4eb814d1b0f5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" Jul 07 14:10:24.181445 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.181160 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54bw\" (UniqueName: \"kubernetes.io/projected/c520cb66-9856-40eb-981c-ad7a9687928a-kube-api-access-n54bw\") pod \"service-ca-operator-5d46d9ffd4-nrjbq\" (UID: \"c520cb66-9856-40eb-981c-ad7a9687928a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" Jul 07 14:10:24.181445 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.181223 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98855958-0a9b-47ca-93a9-8b82df6c7ee6-trusted-ca\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.181445 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:24.181248 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 14:10:24.181445 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.181256 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-w76w5\" (UID: \"fd5fab7c-5de3-4f92-85a9-863fd04af85d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:24.181445 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.181305 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-certificates\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.181445 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.181332 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98855958-0a9b-47ca-93a9-8b82df6c7ee6-installation-pull-secrets\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.181445 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.181363 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98855958-0a9b-47ca-93a9-8b82df6c7ee6-image-registry-private-configuration\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.181445 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.181439 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c520cb66-9856-40eb-981c-ad7a9687928a-config\") pod \"service-ca-operator-5d46d9ffd4-nrjbq\" (UID: \"c520cb66-9856-40eb-981c-ad7a9687928a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" Jul 07 14:10:24.181805 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:24.181473 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 14:10:24.181805 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:24.181486 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls podName:1dd48565-d6eb-406e-beb0-4eb814d1b0f5 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:24.681466169 +0000 UTC m=+183.113708657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-dhvrh" (UID: "1dd48565-d6eb-406e-beb0-4eb814d1b0f5") : secret "samples-operator-tls" not found Jul 07 14:10:24.181805 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:24.181523 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls podName:fd5fab7c-5de3-4f92-85a9-863fd04af85d nodeName:}" failed. No retries permitted until 2026-07-07 14:10:24.681509891 +0000 UTC m=+183.113752385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-w76w5" (UID: "fd5fab7c-5de3-4f92-85a9-863fd04af85d") : secret "cluster-monitoring-operator-tls" not found Jul 07 14:10:24.181805 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.181774 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fd5fab7c-5de3-4f92-85a9-863fd04af85d-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-w76w5\" (UID: \"fd5fab7c-5de3-4f92-85a9-863fd04af85d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:24.183436 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.183414 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c520cb66-9856-40eb-981c-ad7a9687928a-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-nrjbq\" (UID: \"c520cb66-9856-40eb-981c-ad7a9687928a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" Jul 07 14:10:24.196137 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.196107 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n54bw\" (UniqueName: \"kubernetes.io/projected/c520cb66-9856-40eb-981c-ad7a9687928a-kube-api-access-n54bw\") pod \"service-ca-operator-5d46d9ffd4-nrjbq\" (UID: \"c520cb66-9856-40eb-981c-ad7a9687928a\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" Jul 07 14:10:24.197516 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.197496 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpq2p\" (UniqueName: \"kubernetes.io/projected/fd5fab7c-5de3-4f92-85a9-863fd04af85d-kube-api-access-hpq2p\") pod \"cluster-monitoring-operator-64786d9cbb-w76w5\" (UID: \"fd5fab7c-5de3-4f92-85a9-863fd04af85d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:24.197605 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.197578 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvnq2\" (UniqueName: \"kubernetes.io/projected/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-kube-api-access-mvnq2\") pod \"cluster-samples-operator-56fcf86bf-dhvrh\" (UID: \"1dd48565-d6eb-406e-beb0-4eb814d1b0f5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" Jul 07 14:10:24.282669 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.282589 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.282669 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.282624 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.282669 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.282642 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgxkn\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-kube-api-access-pgxkn\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.282952 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.282782 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98855958-0a9b-47ca-93a9-8b82df6c7ee6-ca-trust-extracted\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.282952 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.282932 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgvbv\" (UniqueName: \"kubernetes.io/projected/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-kube-api-access-bgvbv\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.283066 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.282969 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-bound-sa-token\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.283123 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.283064 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98855958-0a9b-47ca-93a9-8b82df6c7ee6-trusted-ca\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.283123 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.283106 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-certificates\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.283222 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.283131 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98855958-0a9b-47ca-93a9-8b82df6c7ee6-installation-pull-secrets\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.283222 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.283160 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98855958-0a9b-47ca-93a9-8b82df6c7ee6-image-registry-private-configuration\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.283222 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.283167 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98855958-0a9b-47ca-93a9-8b82df6c7ee6-ca-trust-extracted\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.283222 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.283200 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-tmp\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.283445 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.283231 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-serving-cert\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.283445 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.283254 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-snapshots\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.283445 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.283297 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.283445 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:24.283392 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:10:24.283445 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:24.283404 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68c89555f7-pxm8l: secret "image-registry-tls" not found Jul 07 14:10:24.283695 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:24.283462 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls podName:98855958-0a9b-47ca-93a9-8b82df6c7ee6 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:24.783445015 +0000 UTC m=+183.215687495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls") pod "image-registry-68c89555f7-pxm8l" (UID: "98855958-0a9b-47ca-93a9-8b82df6c7ee6") : secret "image-registry-tls" not found Jul 07 14:10:24.283695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.283644 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.283813 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.283708 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-certificates\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.283813 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.283198 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.283917 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.283873 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-tmp\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.283974 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.283957 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98855958-0a9b-47ca-93a9-8b82df6c7ee6-trusted-ca\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.284318 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.284293 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-snapshots\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.285796 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.285773 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" Jul 07 14:10:24.286058 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.286038 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-serving-cert\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.286333 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.286312 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98855958-0a9b-47ca-93a9-8b82df6c7ee6-image-registry-private-configuration\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.286387 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.286318 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98855958-0a9b-47ca-93a9-8b82df6c7ee6-installation-pull-secrets\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.295655 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.294130 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgxkn\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-kube-api-access-pgxkn\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.295816 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.295785 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:24.296238 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.296215 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-bound-sa-token\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.296857 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.296833 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgvbv\" (UniqueName: \"kubernetes.io/projected/b9a6eef4-26d4-4d5c-a1ea-6d71968d1116-kube-api-access-bgvbv\") pod \"insights-operator-5c847dcfc4-hwj87\" (UID: \"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116\") " pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.380890 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.380855 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" Jul 07 14:10:24.394025 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.393998 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:24.414715 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.414553 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq"] Jul 07 14:10:24.417207 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:10:24.417182 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc520cb66_9856_40eb_981c_ad7a9687928a.slice/crio-bb74970209b10af1880b54c1b511d1bd83908c1533727f63b1a5db2ef7e4be2e WatchSource:0}: Error finding container bb74970209b10af1880b54c1b511d1bd83908c1533727f63b1a5db2ef7e4be2e: Status 404 returned error can't find the container with id bb74970209b10af1880b54c1b511d1bd83908c1533727f63b1a5db2ef7e4be2e Jul 07 14:10:24.506935 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.506906 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-hwj87"] Jul 07 14:10:24.510327 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:10:24.510295 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a6eef4_26d4_4d5c_a1ea_6d71968d1116.slice/crio-9236ab370f212379af86e86c8e8f2001304d893b15aad14a96be8c1689e56c53 WatchSource:0}: Error finding container 9236ab370f212379af86e86c8e8f2001304d893b15aad14a96be8c1689e56c53: Status 404 returned error can't find the container with id 9236ab370f212379af86e86c8e8f2001304d893b15aad14a96be8c1689e56c53 Jul 07 14:10:24.687400 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.687352 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-dhvrh\" (UID: \"1dd48565-d6eb-406e-beb0-4eb814d1b0f5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" Jul 07 14:10:24.687582 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.687421 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-w76w5\" (UID: \"fd5fab7c-5de3-4f92-85a9-863fd04af85d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:24.687582 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:24.687498 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 14:10:24.687582 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:24.687565 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls podName:1dd48565-d6eb-406e-beb0-4eb814d1b0f5 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:25.687548956 +0000 UTC m=+184.119791435 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-dhvrh" (UID: "1dd48565-d6eb-406e-beb0-4eb814d1b0f5") : secret "samples-operator-tls" not found Jul 07 14:10:24.687582 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:24.687569 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 14:10:24.687724 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:24.687615 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls podName:fd5fab7c-5de3-4f92-85a9-863fd04af85d nodeName:}" failed. No retries permitted until 2026-07-07 14:10:25.687600151 +0000 UTC m=+184.119842630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-w76w5" (UID: "fd5fab7c-5de3-4f92-85a9-863fd04af85d") : secret "cluster-monitoring-operator-tls" not found Jul 07 14:10:24.736657 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.736618 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" event={"ID":"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116","Type":"ContainerStarted","Data":"9236ab370f212379af86e86c8e8f2001304d893b15aad14a96be8c1689e56c53"} Jul 07 14:10:24.737589 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.737566 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" event={"ID":"c520cb66-9856-40eb-981c-ad7a9687928a","Type":"ContainerStarted","Data":"bb74970209b10af1880b54c1b511d1bd83908c1533727f63b1a5db2ef7e4be2e"} Jul 07 14:10:24.788176 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:24.788141 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:24.788317 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:24.788292 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:10:24.788317 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:24.788307 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68c89555f7-pxm8l: secret "image-registry-tls" not found Jul 07 14:10:24.788384 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:24.788356 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls podName:98855958-0a9b-47ca-93a9-8b82df6c7ee6 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:25.788342126 +0000 UTC m=+184.220584606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls") pod "image-registry-68c89555f7-pxm8l" (UID: "98855958-0a9b-47ca-93a9-8b82df6c7ee6") : secret "image-registry-tls" not found Jul 07 14:10:25.693655 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:25.693617 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-dhvrh\" (UID: \"1dd48565-d6eb-406e-beb0-4eb814d1b0f5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" Jul 07 14:10:25.694143 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:25.693696 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-w76w5\" (UID: \"fd5fab7c-5de3-4f92-85a9-863fd04af85d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:25.694143 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:25.693797 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 14:10:25.694143 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:25.693881 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls podName:1dd48565-d6eb-406e-beb0-4eb814d1b0f5 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:27.693859523 +0000 UTC m=+186.126102022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-dhvrh" (UID: "1dd48565-d6eb-406e-beb0-4eb814d1b0f5") : secret "samples-operator-tls" not found Jul 07 14:10:25.694143 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:25.693802 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 14:10:25.694143 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:25.693990 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls podName:fd5fab7c-5de3-4f92-85a9-863fd04af85d nodeName:}" failed. No retries permitted until 2026-07-07 14:10:27.693972913 +0000 UTC m=+186.126215404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-w76w5" (UID: "fd5fab7c-5de3-4f92-85a9-863fd04af85d") : secret "cluster-monitoring-operator-tls" not found Jul 07 14:10:25.794656 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:25.794619 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:25.794813 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:25.794738 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:10:25.794813 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:25.794751 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68c89555f7-pxm8l: secret "image-registry-tls" not found Jul 07 14:10:25.794813 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:25.794808 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls podName:98855958-0a9b-47ca-93a9-8b82df6c7ee6 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:27.794793813 +0000 UTC m=+186.227036292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls") pod "image-registry-68c89555f7-pxm8l" (UID: "98855958-0a9b-47ca-93a9-8b82df6c7ee6") : secret "image-registry-tls" not found Jul 07 14:10:26.371791 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:26.371753 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:26.473100 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:26.473069 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:26.744332 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:26.744305 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" event={"ID":"c520cb66-9856-40eb-981c-ad7a9687928a","Type":"ContainerStarted","Data":"33b9d9730daf063e4f9ad566841d85dac8c20a1f7f7372b0dce5a7b31760d48a"} Jul 07 14:10:26.760036 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:26.759959 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" podStartSLOduration=1.502662885 podStartE2EDuration="3.759943964s" totalImagesPullingTime="2.257281079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:26.759456812 +0000 UTC m=+185.191699316" watchObservedRunningTime="2026-07-07 14:10:26.759943964 +0000 UTC m=+185.192186472" Jul 07 14:10:27.709862 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:27.709767 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-dhvrh\" (UID: \"1dd48565-d6eb-406e-beb0-4eb814d1b0f5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" Jul 07 14:10:27.709862 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:27.709825 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-w76w5\" (UID: \"fd5fab7c-5de3-4f92-85a9-863fd04af85d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:27.710079 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:27.709912 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 14:10:27.710079 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:27.709938 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 14:10:27.710079 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:27.709981 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls podName:1dd48565-d6eb-406e-beb0-4eb814d1b0f5 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:31.709960264 +0000 UTC m=+190.142202744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-dhvrh" (UID: "1dd48565-d6eb-406e-beb0-4eb814d1b0f5") : secret "samples-operator-tls" not found Jul 07 14:10:27.710079 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:27.710009 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls podName:fd5fab7c-5de3-4f92-85a9-863fd04af85d nodeName:}" failed. No retries permitted until 2026-07-07 14:10:31.709996841 +0000 UTC m=+190.142239325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-w76w5" (UID: "fd5fab7c-5de3-4f92-85a9-863fd04af85d") : secret "cluster-monitoring-operator-tls" not found Jul 07 14:10:27.748211 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:27.748178 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" event={"ID":"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116","Type":"ContainerStarted","Data":"db0c78e66027eff53ae3034cbe6d076cb74ceb725bc0370df316be4f632e8566"} Jul 07 14:10:27.764277 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:27.764231 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" podStartSLOduration=0.923544504 podStartE2EDuration="3.764216243s" totalImagesPullingTime="2.840671739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:27.763358986 +0000 UTC m=+186.195601492" watchObservedRunningTime="2026-07-07 14:10:27.764216243 +0000 UTC m=+186.196458744" Jul 07 14:10:27.811147 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:27.811107 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:27.811547 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:27.811371 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:10:27.811547 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:27.811395 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68c89555f7-pxm8l: secret "image-registry-tls" not found Jul 07 14:10:27.811547 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:27.811453 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls podName:98855958-0a9b-47ca-93a9-8b82df6c7ee6 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:31.811435005 +0000 UTC m=+190.243677503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls") pod "image-registry-68c89555f7-pxm8l" (UID: "98855958-0a9b-47ca-93a9-8b82df6c7ee6") : secret "image-registry-tls" not found Jul 07 14:10:28.209800 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.209767 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:28.317498 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.317468 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:28.369850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.369819 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-phxnw"] Jul 07 14:10:28.372755 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.372735 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-phxnw" Jul 07 14:10:28.376424 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.376396 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-65fcl\"" Jul 07 14:10:28.376596 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.376407 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 07 14:10:28.377195 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.377178 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 07 14:10:28.386156 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.386132 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-phxnw"] Jul 07 14:10:28.387029 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.387011 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7xj79"] Jul 07 14:10:28.389278 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.389244 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:28.391290 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.391248 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vkw4d\"" Jul 07 14:10:28.391569 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.391549 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 14:10:28.392084 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.392063 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 14:10:28.405127 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.402594 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7xj79"] Jul 07 14:10:28.416151 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.416120 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbnws\" (UniqueName: \"kubernetes.io/projected/aed7476c-67d4-4194-ba84-fa5b9909a0ce-kube-api-access-pbnws\") pod \"migrator-d57d58cc9-phxnw\" (UID: \"aed7476c-67d4-4194-ba84-fa5b9909a0ce\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-phxnw" Jul 07 14:10:28.416296 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.416180 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:28.416296 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.416237 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:28.416414 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.416336 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4961cb11-50bf-4289-8a88-4100c26959d6-data-volume\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:28.416414 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.416376 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4961cb11-50bf-4289-8a88-4100c26959d6-crio-socket\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:28.416497 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.416424 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s6p6\" (UniqueName: \"kubernetes.io/projected/4961cb11-50bf-4289-8a88-4100c26959d6-kube-api-access-9s6p6\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:28.508974 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.508890 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:28.516969 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.516945 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4961cb11-50bf-4289-8a88-4100c26959d6-crio-socket\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:28.517053 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.516995 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s6p6\" (UniqueName: \"kubernetes.io/projected/4961cb11-50bf-4289-8a88-4100c26959d6-kube-api-access-9s6p6\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:28.517089 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.517071 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4961cb11-50bf-4289-8a88-4100c26959d6-crio-socket\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:28.517175 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.517157 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbnws\" (UniqueName: \"kubernetes.io/projected/aed7476c-67d4-4194-ba84-fa5b9909a0ce-kube-api-access-pbnws\") pod \"migrator-d57d58cc9-phxnw\" (UID: \"aed7476c-67d4-4194-ba84-fa5b9909a0ce\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-phxnw" Jul 07 14:10:28.517212 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.517192 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:28.517251 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.517236 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:28.517320 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.517307 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4961cb11-50bf-4289-8a88-4100c26959d6-data-volume\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:28.517381 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:28.517361 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:10:28.517429 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:28.517376 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:10:28.517429 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:28.517428 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls podName:4961cb11-50bf-4289-8a88-4100c26959d6 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:29.017416298 +0000 UTC m=+187.449658778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7xj79" (UID: "4961cb11-50bf-4289-8a88-4100c26959d6") : secret "insights-runtime-extractor-tls" not found Jul 07 14:10:28.517529 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:28.517441 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm podName:4961cb11-50bf-4289-8a88-4100c26959d6 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:29.017435244 +0000 UTC m=+187.449677723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7xj79" (UID: "4961cb11-50bf-4289-8a88-4100c26959d6") : configmap "kube-rbac-proxy" not found Jul 07 14:10:28.517647 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.517627 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4961cb11-50bf-4289-8a88-4100c26959d6-data-volume\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:28.526391 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.526371 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbnws\" (UniqueName: \"kubernetes.io/projected/aed7476c-67d4-4194-ba84-fa5b9909a0ce-kube-api-access-pbnws\") pod \"migrator-d57d58cc9-phxnw\" (UID: \"aed7476c-67d4-4194-ba84-fa5b9909a0ce\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-phxnw" Jul 07 14:10:28.526634 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.526618 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s6p6\" (UniqueName: \"kubernetes.io/projected/4961cb11-50bf-4289-8a88-4100c26959d6-kube-api-access-9s6p6\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:28.608746 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.608701 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:28.682885 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.682849 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-phxnw" Jul 07 14:10:28.806437 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.806403 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-phxnw"] Jul 07 14:10:28.809889 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:10:28.809865 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed7476c_67d4_4194_ba84_fa5b9909a0ce.slice/crio-8594acc11a77cc5b3e9c10bb95beff4061de5f5151a38647ebb953c780705432 WatchSource:0}: Error finding container 8594acc11a77cc5b3e9c10bb95beff4061de5f5151a38647ebb953c780705432: Status 404 returned error can't find the container with id 8594acc11a77cc5b3e9c10bb95beff4061de5f5151a38647ebb953c780705432 Jul 07 14:10:28.909086 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.909058 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:28.959219 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:28.959194 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:29.021553 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:29.021496 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:29.021660 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:29.021598 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:29.021719 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:29.021661 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:10:29.021719 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:29.021666 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:10:29.021821 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:29.021735 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls podName:4961cb11-50bf-4289-8a88-4100c26959d6 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:30.021714541 +0000 UTC m=+188.453957025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7xj79" (UID: "4961cb11-50bf-4289-8a88-4100c26959d6") : secret "insights-runtime-extractor-tls" not found Jul 07 14:10:29.021821 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:29.021757 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm podName:4961cb11-50bf-4289-8a88-4100c26959d6 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:30.021744901 +0000 UTC m=+188.453987382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7xj79" (UID: "4961cb11-50bf-4289-8a88-4100c26959d6") : configmap "kube-rbac-proxy" not found Jul 07 14:10:29.754684 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:29.754644 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-phxnw" event={"ID":"aed7476c-67d4-4194-ba84-fa5b9909a0ce","Type":"ContainerStarted","Data":"8594acc11a77cc5b3e9c10bb95beff4061de5f5151a38647ebb953c780705432"} Jul 07 14:10:30.030468 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:30.030403 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:30.030468 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:30.030442 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:30.030839 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:30.030533 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:10:30.030839 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:30.030551 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:10:30.030839 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:30.030597 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm podName:4961cb11-50bf-4289-8a88-4100c26959d6 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:32.030580786 +0000 UTC m=+190.462823269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7xj79" (UID: "4961cb11-50bf-4289-8a88-4100c26959d6") : configmap "kube-rbac-proxy" not found Jul 07 14:10:30.030839 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:30.030611 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls podName:4961cb11-50bf-4289-8a88-4100c26959d6 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:32.030604726 +0000 UTC m=+190.462847206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7xj79" (UID: "4961cb11-50bf-4289-8a88-4100c26959d6") : secret "insights-runtime-extractor-tls" not found Jul 07 14:10:30.357693 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:30.357668 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6bc6n_1ac773f2-d330-4140-97aa-8ac19ee4d555/dns-node-resolver/0.log" Jul 07 14:10:30.536414 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:30.536383 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:30.589487 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:30.589458 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:30.759482 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:30.759408 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-phxnw" event={"ID":"aed7476c-67d4-4194-ba84-fa5b9909a0ce","Type":"ContainerStarted","Data":"cf9c3efe84971124eb011418dd6725f5c78cd7bf4bdc65fd824f1411a9a4c8fe"} Jul 07 14:10:30.759482 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:30.759455 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-phxnw" event={"ID":"aed7476c-67d4-4194-ba84-fa5b9909a0ce","Type":"ContainerStarted","Data":"29784b61690ecff8e82b06263fcadc33628d8ee63336c31d536a2cc3000ba895"} Jul 07 14:10:30.773939 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:30.773894 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-phxnw" podStartSLOduration=1.756306683 podStartE2EDuration="2.773876165s" totalImagesPullingTime="1.017569482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:30.773462685 +0000 UTC m=+189.205705187" watchObservedRunningTime="2026-07-07 14:10:30.773876165 +0000 UTC m=+189.206118679" Jul 07 14:10:30.853953 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:30.853911 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:30.904960 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:30.904926 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:31.158314 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:31.158286 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5kt5s_1053d78f-a5ae-4acd-9a99-32beb279bbb4/node-ca/0.log" Jul 07 14:10:31.548945 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:31.548916 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:31.748447 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:31.745449 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-dhvrh\" (UID: \"1dd48565-d6eb-406e-beb0-4eb814d1b0f5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" Jul 07 14:10:31.748447 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:31.745551 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-w76w5\" (UID: \"fd5fab7c-5de3-4f92-85a9-863fd04af85d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:31.748447 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:31.745735 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 14:10:31.748447 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:31.745835 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls podName:fd5fab7c-5de3-4f92-85a9-863fd04af85d nodeName:}" failed. No retries permitted until 2026-07-07 14:10:39.74581638 +0000 UTC m=+198.178058860 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-w76w5" (UID: "fd5fab7c-5de3-4f92-85a9-863fd04af85d") : secret "cluster-monitoring-operator-tls" not found Jul 07 14:10:31.748447 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:31.746352 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 07 14:10:31.748447 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:31.746412 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls podName:1dd48565-d6eb-406e-beb0-4eb814d1b0f5 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:39.746395691 +0000 UTC m=+198.178638187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-dhvrh" (UID: "1dd48565-d6eb-406e-beb0-4eb814d1b0f5") : secret "samples-operator-tls" not found Jul 07 14:10:31.846680 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:31.846591 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:31.846823 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:31.846752 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 14:10:31.846823 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:31.846763 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68c89555f7-pxm8l: secret "image-registry-tls" not found Jul 07 14:10:31.846907 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:31.846835 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls podName:98855958-0a9b-47ca-93a9-8b82df6c7ee6 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:39.84682209 +0000 UTC m=+198.279064570 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls") pod "image-registry-68c89555f7-pxm8l" (UID: "98855958-0a9b-47ca-93a9-8b82df6c7ee6") : secret "image-registry-tls" not found Jul 07 14:10:31.957619 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:31.957588 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5cgb6_a334bf52-3bf3-4862-8680-d4e89f3d28bc/serve-healthcheck-canary/0.log" Jul 07 14:10:32.048952 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:32.048922 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:32.049100 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:32.048969 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:32.049100 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:32.049041 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:10:32.049164 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:32.049130 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:10:32.049209 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:32.049131 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm podName:4961cb11-50bf-4289-8a88-4100c26959d6 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:36.049109035 +0000 UTC m=+194.481351515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7xj79" (UID: "4961cb11-50bf-4289-8a88-4100c26959d6") : configmap "kube-rbac-proxy" not found Jul 07 14:10:32.049209 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:32.049185 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls podName:4961cb11-50bf-4289-8a88-4100c26959d6 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:36.049171663 +0000 UTC m=+194.481414142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7xj79" (UID: "4961cb11-50bf-4289-8a88-4100c26959d6") : secret "insights-runtime-extractor-tls" not found Jul 07 14:10:33.973822 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:33.973789 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:35.693582 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:35.693541 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:35.894283 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:35.894233 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:35.995203 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:35.995125 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:36.082402 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:36.082376 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:36.082527 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:36.082417 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:36.082527 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:36.082486 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 14:10:36.082527 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:36.082505 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:10:36.082649 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:36.082554 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm podName:4961cb11-50bf-4289-8a88-4100c26959d6 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:44.082539701 +0000 UTC m=+202.514782181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7xj79" (UID: "4961cb11-50bf-4289-8a88-4100c26959d6") : configmap "kube-rbac-proxy" not found Jul 07 14:10:36.082649 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:36.082568 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls podName:4961cb11-50bf-4289-8a88-4100c26959d6 nodeName:}" failed. No retries permitted until 2026-07-07 14:10:44.082561877 +0000 UTC m=+202.514804357 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7xj79" (UID: "4961cb11-50bf-4289-8a88-4100c26959d6") : secret "insights-runtime-extractor-tls" not found Jul 07 14:10:36.096257 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:36.096233 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:36.201056 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:36.201031 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:36.295903 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:36.295873 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:36.396980 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:36.396949 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:37.583307 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:37.583255 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:37.779830 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:37.779799 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:37.981592 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:37.981504 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:38.180997 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:38.180963 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:38.312926 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:38.312895 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:38.388874 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:38.388842 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:39.521894 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:39.521862 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:39.623335 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:39.623304 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:39.748562 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:39.748529 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:39.805238 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:39.805208 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:39.811347 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:39.811324 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-dhvrh\" (UID: \"1dd48565-d6eb-406e-beb0-4eb814d1b0f5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" Jul 07 14:10:39.811475 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:39.811392 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-w76w5\" (UID: \"fd5fab7c-5de3-4f92-85a9-863fd04af85d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:39.811556 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:39.811541 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 07 14:10:39.811620 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:39.811611 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls podName:fd5fab7c-5de3-4f92-85a9-863fd04af85d nodeName:}" failed. No retries permitted until 2026-07-07 14:10:55.811596436 +0000 UTC m=+214.243838916 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-w76w5" (UID: "fd5fab7c-5de3-4f92-85a9-863fd04af85d") : secret "cluster-monitoring-operator-tls" not found Jul 07 14:10:39.813858 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:39.813839 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1dd48565-d6eb-406e-beb0-4eb814d1b0f5-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-dhvrh\" (UID: \"1dd48565-d6eb-406e-beb0-4eb814d1b0f5\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" Jul 07 14:10:39.890541 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:39.890499 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" Jul 07 14:10:39.911984 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:39.911950 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:39.914464 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:39.914436 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls\") pod \"image-registry-68c89555f7-pxm8l\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:39.986664 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:39.986617 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:39.991203 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:39.988386 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:40.020946 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:40.020911 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh"] Jul 07 14:10:40.121476 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:40.121449 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68c89555f7-pxm8l"] Jul 07 14:10:40.124189 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:10:40.124159 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98855958_0a9b_47ca_93a9_8b82df6c7ee6.slice/crio-49e2b35830cf810573c828c392646820d86407c37dc7813c6499ba0f787eccc9 WatchSource:0}: Error finding container 49e2b35830cf810573c828c392646820d86407c37dc7813c6499ba0f787eccc9: Status 404 returned error can't find the container with id 49e2b35830cf810573c828c392646820d86407c37dc7813c6499ba0f787eccc9 Jul 07 14:10:40.186039 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:40.186015 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:40.379890 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:40.379815 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:40.788855 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:40.788775 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" event={"ID":"98855958-0a9b-47ca-93a9-8b82df6c7ee6","Type":"ContainerStarted","Data":"f6430c9f5f71e388fad7b144189a0d331b95c9c0c7e7a53512c35dbd65746c47"} Jul 07 14:10:40.788855 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:40.788820 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" event={"ID":"98855958-0a9b-47ca-93a9-8b82df6c7ee6","Type":"ContainerStarted","Data":"49e2b35830cf810573c828c392646820d86407c37dc7813c6499ba0f787eccc9"} Jul 07 14:10:40.789328 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:40.788889 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:10:40.789972 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:40.789945 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" event={"ID":"1dd48565-d6eb-406e-beb0-4eb814d1b0f5","Type":"ContainerStarted","Data":"2b1225a3a8fe6c9d9c7dc3ad17922499f1b158bbcfdd4b412db39459a238132d"} Jul 07 14:10:40.808124 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:40.808083 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" podStartSLOduration=16.80806794 podStartE2EDuration="16.80806794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:40.807427283 +0000 UTC m=+199.239669786" watchObservedRunningTime="2026-07-07 14:10:40.80806794 +0000 UTC m=+199.240310441" Jul 07 14:10:41.181682 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:41.181648 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:41.382100 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:41.382067 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:41.440539 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:41.440466 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:42.162629 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:42.162600 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:42.258458 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:42.258421 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:42.612552 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:42.612510 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:42.781516 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:42.781420 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:42.797521 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:42.797495 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" event={"ID":"1dd48565-d6eb-406e-beb0-4eb814d1b0f5","Type":"ContainerStarted","Data":"f8ee4302f2582eaba0dc84d2d35117ebcba653365d3e7d26db99b250119604f5"} Jul 07 14:10:42.797655 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:42.797545 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" event={"ID":"1dd48565-d6eb-406e-beb0-4eb814d1b0f5","Type":"ContainerStarted","Data":"2b4f761e47ce11cd0e8ec478a4c2c638c234feb92a995bd156a51879f3586e38"} Jul 07 14:10:42.813881 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:42.813836 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-dhvrh" podStartSLOduration=17.528483954 podStartE2EDuration="19.813819511s" totalImagesPullingTime="2.285335557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:42.813076192 +0000 UTC m=+201.245318694" watchObservedRunningTime="2026-07-07 14:10:42.813819511 +0000 UTC m=+201.246062014" Jul 07 14:10:42.983925 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:42.983890 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:43.181604 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:43.181577 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:44.148737 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:44.148685 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:44.148948 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:44.148747 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:44.148948 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:44.148893 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 14:10:44.149063 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:10:44.148960 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls podName:4961cb11-50bf-4289-8a88-4100c26959d6 nodeName:}" failed. No retries permitted until 2026-07-07 14:11:00.148941781 +0000 UTC m=+218.581184278 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7xj79" (UID: "4961cb11-50bf-4289-8a88-4100c26959d6") : secret "insights-runtime-extractor-tls" not found Jul 07 14:10:44.149217 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:44.149193 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4961cb11-50bf-4289-8a88-4100c26959d6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:10:44.389801 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:44.389761 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:44.588198 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:44.588147 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:44.783559 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:44.783523 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:44.984387 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:44.984299 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:45.190394 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:45.190358 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:45.992898 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:45.992866 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:46.184049 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:46.184014 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:46.387948 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:46.387918 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:49.959017 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:49.958979 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68c89555f7-pxm8l"] Jul 07 14:10:49.988246 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:49.988216 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:49.999113 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:49.999087 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d85bcc555-qb8tv"] Jul 07 14:10:50.002020 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.002003 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.014206 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.014185 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d85bcc555-qb8tv"] Jul 07 14:10:50.080596 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.080560 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:50.093616 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.093590 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff734b15-88fd-402d-8c78-0a60cb48c5d8-installation-pull-secrets\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.093708 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.093620 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w8s5\" (UniqueName: \"kubernetes.io/projected/ff734b15-88fd-402d-8c78-0a60cb48c5d8-kube-api-access-9w8s5\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.093708 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.093650 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff734b15-88fd-402d-8c78-0a60cb48c5d8-registry-tls\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.093782 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.093722 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff734b15-88fd-402d-8c78-0a60cb48c5d8-image-registry-private-configuration\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.093782 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.093762 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff734b15-88fd-402d-8c78-0a60cb48c5d8-ca-trust-extracted\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.093857 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.093843 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff734b15-88fd-402d-8c78-0a60cb48c5d8-trusted-ca\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.093892 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.093870 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff734b15-88fd-402d-8c78-0a60cb48c5d8-bound-sa-token\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.093966 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.093949 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff734b15-88fd-402d-8c78-0a60cb48c5d8-registry-certificates\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.183388 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.183359 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:50.194713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.194690 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff734b15-88fd-402d-8c78-0a60cb48c5d8-trusted-ca\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.194713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.194718 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff734b15-88fd-402d-8c78-0a60cb48c5d8-bound-sa-token\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.194940 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.194749 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff734b15-88fd-402d-8c78-0a60cb48c5d8-registry-certificates\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.194940 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.194774 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff734b15-88fd-402d-8c78-0a60cb48c5d8-installation-pull-secrets\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.194940 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.194791 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w8s5\" (UniqueName: \"kubernetes.io/projected/ff734b15-88fd-402d-8c78-0a60cb48c5d8-kube-api-access-9w8s5\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.194940 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.194811 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff734b15-88fd-402d-8c78-0a60cb48c5d8-registry-tls\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.194940 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.194851 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff734b15-88fd-402d-8c78-0a60cb48c5d8-image-registry-private-configuration\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.194940 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.194876 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff734b15-88fd-402d-8c78-0a60cb48c5d8-ca-trust-extracted\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.195467 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.195435 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff734b15-88fd-402d-8c78-0a60cb48c5d8-ca-trust-extracted\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.195868 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.195841 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff734b15-88fd-402d-8c78-0a60cb48c5d8-registry-certificates\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.196247 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.196220 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff734b15-88fd-402d-8c78-0a60cb48c5d8-trusted-ca\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.197577 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.197547 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff734b15-88fd-402d-8c78-0a60cb48c5d8-installation-pull-secrets\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.197663 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.197573 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff734b15-88fd-402d-8c78-0a60cb48c5d8-registry-tls\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.197703 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.197661 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff734b15-88fd-402d-8c78-0a60cb48c5d8-image-registry-private-configuration\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.203558 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.203536 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w8s5\" (UniqueName: \"kubernetes.io/projected/ff734b15-88fd-402d-8c78-0a60cb48c5d8-kube-api-access-9w8s5\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.203737 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.203719 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff734b15-88fd-402d-8c78-0a60cb48c5d8-bound-sa-token\") pod \"image-registry-7d85bcc555-qb8tv\" (UID: \"ff734b15-88fd-402d-8c78-0a60cb48c5d8\") " pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.311683 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.311656 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.440226 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.440198 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d85bcc555-qb8tv"] Jul 07 14:10:50.443009 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:10:50.442986 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff734b15_88fd_402d_8c78_0a60cb48c5d8.slice/crio-9fd2f2ba7450b5981c11fb075af084249d23efb33a7b9dd5d7170a361c054f39 WatchSource:0}: Error finding container 9fd2f2ba7450b5981c11fb075af084249d23efb33a7b9dd5d7170a361c054f39: Status 404 returned error can't find the container with id 9fd2f2ba7450b5981c11fb075af084249d23efb33a7b9dd5d7170a361c054f39 Jul 07 14:10:50.826090 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.826058 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" event={"ID":"ff734b15-88fd-402d-8c78-0a60cb48c5d8","Type":"ContainerStarted","Data":"ae679b40196ca588eaf0e30bda704837a1326b686264a3c7f0c3c12828dce0ef"} Jul 07 14:10:50.826294 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.826109 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" event={"ID":"ff734b15-88fd-402d-8c78-0a60cb48c5d8","Type":"ContainerStarted","Data":"9fd2f2ba7450b5981c11fb075af084249d23efb33a7b9dd5d7170a361c054f39"} Jul 07 14:10:50.826294 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.826182 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:10:50.845315 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:50.845248 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" podStartSLOduration=1.845231244 podStartE2EDuration="1.845231244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:50.84448676 +0000 UTC m=+209.276729261" watchObservedRunningTime="2026-07-07 14:10:50.845231244 +0000 UTC m=+209.277473774" Jul 07 14:10:55.834426 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:55.834376 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-w76w5\" (UID: \"fd5fab7c-5de3-4f92-85a9-863fd04af85d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:55.836965 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:55.836934 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fd5fab7c-5de3-4f92-85a9-863fd04af85d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-w76w5\" (UID: \"fd5fab7c-5de3-4f92-85a9-863fd04af85d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:56.080513 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:56.080479 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" Jul 07 14:10:56.207963 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:56.207934 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5"] Jul 07 14:10:56.210752 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:10:56.210727 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5fab7c_5de3_4f92_85a9_863fd04af85d.slice/crio-48051aa6c6b800619821da49c0b2ed84cc2e64f849a9c46ecc99f90a7acc7d5e WatchSource:0}: Error finding container 48051aa6c6b800619821da49c0b2ed84cc2e64f849a9c46ecc99f90a7acc7d5e: Status 404 returned error can't find the container with id 48051aa6c6b800619821da49c0b2ed84cc2e64f849a9c46ecc99f90a7acc7d5e Jul 07 14:10:56.842931 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:56.842892 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" event={"ID":"fd5fab7c-5de3-4f92-85a9-863fd04af85d","Type":"ContainerStarted","Data":"48051aa6c6b800619821da49c0b2ed84cc2e64f849a9c46ecc99f90a7acc7d5e"} Jul 07 14:10:58.416310 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:58.416201 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:58.513367 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:58.513328 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:58.620644 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:58.620606 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:58.682544 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:58.682484 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:58.769952 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:58.769931 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:10:58.850409 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:58.850386 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" event={"ID":"fd5fab7c-5de3-4f92-85a9-863fd04af85d","Type":"ContainerStarted","Data":"4f01285b1c6092ac9d8c0be50445a6dff1b1ff652c07a558d52b3f3a3fd84562"} Jul 07 14:10:58.866440 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:58.866399 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-w76w5" podStartSLOduration=33.995578014 podStartE2EDuration="35.866386799s" totalImagesPullingTime="1.870808785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:10:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:10:58.865891465 +0000 UTC m=+217.298133967" watchObservedRunningTime="2026-07-07 14:10:58.866386799 +0000 UTC m=+217.298629300" Jul 07 14:10:59.964645 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:59.964609 2560 patch_prober.go:28] interesting pod/image-registry-68c89555f7-pxm8l container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 14:10:59.965055 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:10:59.964661 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" podUID="98855958-0a9b-47ca-93a9-8b82df6c7ee6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:11:00.166168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:00.166134 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:11:00.168496 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:00.168467 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4961cb11-50bf-4289-8a88-4100c26959d6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7xj79\" (UID: \"4961cb11-50bf-4289-8a88-4100c26959d6\") " pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:11:00.197530 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:00.197503 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7xj79" Jul 07 14:11:00.324291 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:00.324243 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7xj79"] Jul 07 14:11:00.327287 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:11:00.327236 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4961cb11_50bf_4289_8a88_4100c26959d6.slice/crio-d3518be8a427ff572a4d52927dc3a19ea958da9b6b0407ca9353a0949a0ce1d6 WatchSource:0}: Error finding container d3518be8a427ff572a4d52927dc3a19ea958da9b6b0407ca9353a0949a0ce1d6: Status 404 returned error can't find the container with id d3518be8a427ff572a4d52927dc3a19ea958da9b6b0407ca9353a0949a0ce1d6 Jul 07 14:11:00.652914 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:00.652878 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:00.725123 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:00.725092 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:00.812304 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:00.812257 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:00.857632 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:00.857602 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7xj79" event={"ID":"4961cb11-50bf-4289-8a88-4100c26959d6","Type":"ContainerStarted","Data":"d3518be8a427ff572a4d52927dc3a19ea958da9b6b0407ca9353a0949a0ce1d6"} Jul 07 14:11:04.871983 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:04.871947 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7xj79" event={"ID":"4961cb11-50bf-4289-8a88-4100c26959d6","Type":"ContainerStarted","Data":"e9a5802a852a2d92a53133b4480c696184150bad91e90589884e911c145c696d"} Jul 07 14:11:05.014631 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:05.014593 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:05.877480 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:05.877443 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7xj79" event={"ID":"4961cb11-50bf-4289-8a88-4100c26959d6","Type":"ContainerStarted","Data":"f147bcfe76bb5baf402f782aef2aac37678022d5766d479f63e98aac5913b77c"} Jul 07 14:11:06.056219 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:06.056172 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:06.157551 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:06.157472 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:06.781941 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:06.781898 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:07.010034 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.009999 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:07.045115 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.045091 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nz8cf"] Jul 07 14:11:07.048407 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.048387 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.051203 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.051173 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 14:11:07.051337 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.051220 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 14:11:07.051602 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.051583 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5v7xh\"" Jul 07 14:11:07.051731 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.051711 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 14:11:07.051822 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.051730 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 14:11:07.086405 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.086377 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:07.155533 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.155493 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:07.225594 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.225561 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-wtmp\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.225737 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.225611 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de614e33-c9bd-4ba8-850f-8a49fc2b0607-metrics-client-ca\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.225737 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.225664 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5hm7\" (UniqueName: \"kubernetes.io/projected/de614e33-c9bd-4ba8-850f-8a49fc2b0607-kube-api-access-n5hm7\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.225838 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.225746 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-textfile\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.225838 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.225782 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-tls\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.225838 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.225807 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-accelerators-collector-config\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.225971 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.225910 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.226022 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.225971 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de614e33-c9bd-4ba8-850f-8a49fc2b0607-sys\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.226022 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.225999 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de614e33-c9bd-4ba8-850f-8a49fc2b0607-root\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.268498 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.268461 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:07.327373 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.327298 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-wtmp\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.327373 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.327354 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de614e33-c9bd-4ba8-850f-8a49fc2b0607-metrics-client-ca\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.327568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.327390 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5hm7\" (UniqueName: \"kubernetes.io/projected/de614e33-c9bd-4ba8-850f-8a49fc2b0607-kube-api-access-n5hm7\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.327568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.327421 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-textfile\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.327568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.327444 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-tls\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.327568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.327472 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-accelerators-collector-config\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.327568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.327489 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-wtmp\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.327568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.327551 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.327568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.327572 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de614e33-c9bd-4ba8-850f-8a49fc2b0607-sys\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.327922 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.327588 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de614e33-c9bd-4ba8-850f-8a49fc2b0607-root\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.327922 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.327664 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de614e33-c9bd-4ba8-850f-8a49fc2b0607-root\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.327922 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.327818 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de614e33-c9bd-4ba8-850f-8a49fc2b0607-sys\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.327922 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:11:07.327863 2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 07 14:11:07.328112 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:11:07.327928 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-tls podName:de614e33-c9bd-4ba8-850f-8a49fc2b0607 nodeName:}" failed. No retries permitted until 2026-07-07 14:11:07.827911461 +0000 UTC m=+226.260153945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-tls") pod "node-exporter-nz8cf" (UID: "de614e33-c9bd-4ba8-850f-8a49fc2b0607") : secret "node-exporter-tls" not found Jul 07 14:11:07.328112 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.328037 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de614e33-c9bd-4ba8-850f-8a49fc2b0607-metrics-client-ca\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.328112 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.328088 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-accelerators-collector-config\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.328256 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.328230 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-textfile\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.330546 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.330525 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.338330 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.338305 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5hm7\" (UniqueName: \"kubernetes.io/projected/de614e33-c9bd-4ba8-850f-8a49fc2b0607-kube-api-access-n5hm7\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.386128 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.386083 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:07.471243 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.471185 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:07.570034 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.569993 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:07.670563 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.670494 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:07.831965 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.831930 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-tls\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.834556 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.834531 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de614e33-c9bd-4ba8-850f-8a49fc2b0607-node-exporter-tls\") pod \"node-exporter-nz8cf\" (UID: \"de614e33-c9bd-4ba8-850f-8a49fc2b0607\") " pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.960060 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:07.960038 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nz8cf" Jul 07 14:11:07.971205 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:11:07.971166 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde614e33_c9bd_4ba8_850f_8a49fc2b0607.slice/crio-67a17f061a10d3f88c427e9c4036e88bb5753c669c2a89509be15b0e07fa6d62 WatchSource:0}: Error finding container 67a17f061a10d3f88c427e9c4036e88bb5753c669c2a89509be15b0e07fa6d62: Status 404 returned error can't find the container with id 67a17f061a10d3f88c427e9c4036e88bb5753c669c2a89509be15b0e07fa6d62 Jul 07 14:11:08.032949 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.032910 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:08.113029 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.112994 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:08.117483 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.117461 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:11:08.121892 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.121878 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.124118 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.124075 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 14:11:08.124244 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.124123 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f6wdq\"" Jul 07 14:11:08.124244 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.124161 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 14:11:08.124244 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.124234 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 14:11:08.124432 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.124122 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 14:11:08.124432 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.124345 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 14:11:08.124432 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.124418 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 14:11:08.124648 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.124632 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 14:11:08.124735 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.124670 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 14:11:08.125246 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.125218 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 14:11:08.138107 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.138085 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:11:08.236497 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.236418 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-config-volume\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.236497 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.236451 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlhfp\" (UniqueName: \"kubernetes.io/projected/f5c09135-a711-4e32-8327-84101b838752-kube-api-access-nlhfp\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.236497 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.236493 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-web-config\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.236727 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.236544 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5c09135-a711-4e32-8327-84101b838752-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.236727 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.236618 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c09135-a711-4e32-8327-84101b838752-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.236727 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.236660 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.236727 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.236686 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.236727 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.236709 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.236896 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.236735 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c09135-a711-4e32-8327-84101b838752-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.236896 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.236782 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5c09135-a711-4e32-8327-84101b838752-config-out\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.236896 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.236804 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.236896 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.236833 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.236896 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.236865 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f5c09135-a711-4e32-8327-84101b838752-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.337525 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.337479 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f5c09135-a711-4e32-8327-84101b838752-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.337717 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.337577 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-config-volume\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.337717 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.337599 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlhfp\" (UniqueName: \"kubernetes.io/projected/f5c09135-a711-4e32-8327-84101b838752-kube-api-access-nlhfp\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.337717 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.337648 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-web-config\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.337717 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.337674 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5c09135-a711-4e32-8327-84101b838752-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.337928 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.337719 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c09135-a711-4e32-8327-84101b838752-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.337928 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.337749 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.337928 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.337778 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.337928 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.337803 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.337928 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.337827 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c09135-a711-4e32-8327-84101b838752-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.337928 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.337866 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f5c09135-a711-4e32-8327-84101b838752-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.337928 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.337878 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5c09135-a711-4e32-8327-84101b838752-config-out\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.337928 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.337906 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.338386 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.337936 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.338386 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:11:08.338062 2560 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 07 14:11:08.338386 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:11:08.338117 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-main-tls podName:f5c09135-a711-4e32-8327-84101b838752 nodeName:}" failed. No retries permitted until 2026-07-07 14:11:08.838100487 +0000 UTC m=+227.270342968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f5c09135-a711-4e32-8327-84101b838752") : secret "alertmanager-main-tls" not found Jul 07 14:11:08.339128 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.339102 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c09135-a711-4e32-8327-84101b838752-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.339736 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:11:08.339386 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5c09135-a711-4e32-8327-84101b838752-alertmanager-trusted-ca-bundle podName:f5c09135-a711-4e32-8327-84101b838752 nodeName:}" failed. No retries permitted until 2026-07-07 14:11:08.83936686 +0000 UTC m=+227.271609345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f5c09135-a711-4e32-8327-84101b838752-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "f5c09135-a711-4e32-8327-84101b838752") : configmap references non-existent config key: ca-bundle.crt Jul 07 14:11:08.340956 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.340903 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.341059 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.340960 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-config-volume\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.341059 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.341000 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-web-config\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.341199 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.341171 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5c09135-a711-4e32-8327-84101b838752-config-out\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.341318 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.341231 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5c09135-a711-4e32-8327-84101b838752-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.341370 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.341331 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.343424 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.343404 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.343501 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.343463 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.346340 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.346323 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlhfp\" (UniqueName: \"kubernetes.io/projected/f5c09135-a711-4e32-8327-84101b838752-kube-api-access-nlhfp\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.842313 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.842249 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.842497 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.842397 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c09135-a711-4e32-8327-84101b838752-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.842497 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:11:08.842415 2560 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 07 14:11:08.842497 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:11:08.842496 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-main-tls podName:f5c09135-a711-4e32-8327-84101b838752 nodeName:}" failed. No retries permitted until 2026-07-07 14:11:09.8424741 +0000 UTC m=+228.274716595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f5c09135-a711-4e32-8327-84101b838752") : secret "alertmanager-main-tls" not found Jul 07 14:11:08.843100 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.843080 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c09135-a711-4e32-8327-84101b838752-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:08.889800 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.889754 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nz8cf" event={"ID":"de614e33-c9bd-4ba8-850f-8a49fc2b0607","Type":"ContainerStarted","Data":"67a17f061a10d3f88c427e9c4036e88bb5753c669c2a89509be15b0e07fa6d62"} Jul 07 14:11:08.892192 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.892158 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7xj79" event={"ID":"4961cb11-50bf-4289-8a88-4100c26959d6","Type":"ContainerStarted","Data":"f369cd95a08b67a5c8e10ef22cdcb4641e6094ef03f868dedf3eec053a39993a"} Jul 07 14:11:08.912247 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:08.912187 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7xj79" podStartSLOduration=33.476381923 podStartE2EDuration="40.912164425s" totalImagesPullingTime="7.435782502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:10:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:08.911509315 +0000 UTC m=+227.343751818" watchObservedRunningTime="2026-07-07 14:11:08.912164425 +0000 UTC m=+227.344406931" Jul 07 14:11:09.304850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:09.304815 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:09.850522 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:09.850483 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:09.853038 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:09.853017 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:09.897124 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:09.897088 2560 generic.go:356] "Generic (PLEG): container finished" podID="de614e33-c9bd-4ba8-850f-8a49fc2b0607" containerID="f4295b06f57616ab5aa8fad488cf23ad6a3c708271867e4fba84f9501286f2df" exitCode=0 Jul 07 14:11:09.897303 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:09.897176 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nz8cf" event={"ID":"de614e33-c9bd-4ba8-850f-8a49fc2b0607","Type":"ContainerDied","Data":"f4295b06f57616ab5aa8fad488cf23ad6a3c708271867e4fba84f9501286f2df"} Jul 07 14:11:09.938221 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:09.938185 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:11:09.964845 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:09.964806 2560 patch_prober.go:28] interesting pod/image-registry-68c89555f7-pxm8l container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 14:11:09.964974 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:09.964862 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" podUID="98855958-0a9b-47ca-93a9-8b82df6c7ee6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:11:09.998306 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:09.998142 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:10.072208 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:10.072180 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:10.089050 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:10.089019 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:11:10.093140 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:11:10.093112 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c09135_a711_4e32_8327_84101b838752.slice/crio-638627b838bcbc8d679851ba5b9388486bcf982b734cbf604396f1c6c3948603 WatchSource:0}: Error finding container 638627b838bcbc8d679851ba5b9388486bcf982b734cbf604396f1c6c3948603: Status 404 returned error can't find the container with id 638627b838bcbc8d679851ba5b9388486bcf982b734cbf604396f1c6c3948603 Jul 07 14:11:10.172543 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:10.172486 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:10.271563 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:10.271524 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:10.316446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:10.316418 2560 patch_prober.go:28] interesting pod/image-registry-7d85bcc555-qb8tv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 14:11:10.316790 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:10.316459 2560 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" podUID="ff734b15-88fd-402d-8c78-0a60cb48c5d8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:11:10.902329 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:10.902298 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c09135-a711-4e32-8327-84101b838752","Type":"ContainerStarted","Data":"638627b838bcbc8d679851ba5b9388486bcf982b734cbf604396f1c6c3948603"} Jul 07 14:11:10.904414 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:10.904377 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nz8cf" event={"ID":"de614e33-c9bd-4ba8-850f-8a49fc2b0607","Type":"ContainerStarted","Data":"88fe115d688ba7b377efd9dce7c160db14bc57c7e62d6ffa11e55f17cf968d68"} Jul 07 14:11:10.904551 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:10.904430 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nz8cf" event={"ID":"de614e33-c9bd-4ba8-850f-8a49fc2b0607","Type":"ContainerStarted","Data":"1041ec6cbb67235cd0116a827b33dc194be237c8a7e3382551bc9cc4cfdbb704"} Jul 07 14:11:10.924079 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:10.924040 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nz8cf" podStartSLOduration=2.733985034 podStartE2EDuration="3.924026424s" totalImagesPullingTime="1.12122441s" totalInitContainerRuntime="68.81698ms" isStatefulPod=true podCreationTimestamp="2026-07-07 14:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:10.922220917 +0000 UTC m=+229.354463426" watchObservedRunningTime="2026-07-07 14:11:10.924026424 +0000 UTC m=+229.356268925" Jul 07 14:11:11.441900 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.441875 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-f6c6c6887-mgxfb"] Jul 07 14:11:11.445704 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.445686 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.450161 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.449620 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 07 14:11:11.450161 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.449982 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7do1fh0vp388b\"" Jul 07 14:11:11.450345 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.450154 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 14:11:11.450741 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.450721 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 07 14:11:11.451024 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.451007 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 07 14:11:11.451329 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.451310 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dwsrp\"" Jul 07 14:11:11.456512 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.456487 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f6c6c6887-mgxfb"] Jul 07 14:11:11.566385 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.566350 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/74877d22-3734-41b4-a37c-158de4eabac0-client-ca-bundle\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.566527 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.566400 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/74877d22-3734-41b4-a37c-158de4eabac0-secret-metrics-server-tls\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.566527 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.566461 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/74877d22-3734-41b4-a37c-158de4eabac0-audit-log\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.566527 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.566493 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/74877d22-3734-41b4-a37c-158de4eabac0-secret-metrics-server-client-certs\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.566527 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.566519 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74877d22-3734-41b4-a37c-158de4eabac0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.566648 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.566575 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvk7t\" (UniqueName: \"kubernetes.io/projected/74877d22-3734-41b4-a37c-158de4eabac0-kube-api-access-wvk7t\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.566648 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.566596 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/74877d22-3734-41b4-a37c-158de4eabac0-metrics-server-audit-profiles\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.667641 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.667580 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvk7t\" (UniqueName: \"kubernetes.io/projected/74877d22-3734-41b4-a37c-158de4eabac0-kube-api-access-wvk7t\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.667641 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.667611 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/74877d22-3734-41b4-a37c-158de4eabac0-metrics-server-audit-profiles\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.667755 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.667645 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/74877d22-3734-41b4-a37c-158de4eabac0-client-ca-bundle\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.667755 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.667684 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/74877d22-3734-41b4-a37c-158de4eabac0-secret-metrics-server-tls\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.667755 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.667719 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/74877d22-3734-41b4-a37c-158de4eabac0-audit-log\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.667755 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.667744 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/74877d22-3734-41b4-a37c-158de4eabac0-secret-metrics-server-client-certs\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.667931 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.667769 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74877d22-3734-41b4-a37c-158de4eabac0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.668212 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.668183 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/74877d22-3734-41b4-a37c-158de4eabac0-audit-log\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.668533 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.668474 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74877d22-3734-41b4-a37c-158de4eabac0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.668720 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.668700 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/74877d22-3734-41b4-a37c-158de4eabac0-metrics-server-audit-profiles\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.670223 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.670200 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/74877d22-3734-41b4-a37c-158de4eabac0-client-ca-bundle\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.670338 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.670239 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/74877d22-3734-41b4-a37c-158de4eabac0-secret-metrics-server-tls\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.670394 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.670350 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/74877d22-3734-41b4-a37c-158de4eabac0-secret-metrics-server-client-certs\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.675122 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.675105 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvk7t\" (UniqueName: \"kubernetes.io/projected/74877d22-3734-41b4-a37c-158de4eabac0-kube-api-access-wvk7t\") pod \"metrics-server-f6c6c6887-mgxfb\" (UID: \"74877d22-3734-41b4-a37c-158de4eabac0\") " pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.758110 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.758080 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:11.809577 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.809543 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:11.816249 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.816082 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf"] Jul 07 14:11:11.819861 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.819843 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf" Jul 07 14:11:11.822300 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.822254 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 07 14:11:11.822424 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.822320 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vjqpw\"" Jul 07 14:11:11.827538 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.827516 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf"] Jul 07 14:11:11.833715 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.833693 2560 patch_prober.go:28] interesting pod/image-registry-7d85bcc555-qb8tv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 14:11:11.833803 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.833734 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" podUID="ff734b15-88fd-402d-8c78-0a60cb48c5d8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:11:11.889855 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.889835 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f6c6c6887-mgxfb"] Jul 07 14:11:11.892371 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:11:11.892346 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74877d22_3734_41b4_a37c_158de4eabac0.slice/crio-cf184f5807ba474a802f97caee5bf2d1345039a50c5246bd87b2cfa1f91d9f29 WatchSource:0}: Error finding container cf184f5807ba474a802f97caee5bf2d1345039a50c5246bd87b2cfa1f91d9f29: Status 404 returned error can't find the container with id cf184f5807ba474a802f97caee5bf2d1345039a50c5246bd87b2cfa1f91d9f29 Jul 07 14:11:11.910379 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.910351 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" event={"ID":"74877d22-3734-41b4-a37c-158de4eabac0","Type":"ContainerStarted","Data":"cf184f5807ba474a802f97caee5bf2d1345039a50c5246bd87b2cfa1f91d9f29"} Jul 07 14:11:11.911931 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.911907 2560 generic.go:356] "Generic (PLEG): container finished" podID="f5c09135-a711-4e32-8327-84101b838752" containerID="adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c" exitCode=0 Jul 07 14:11:11.912025 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.911984 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c09135-a711-4e32-8327-84101b838752","Type":"ContainerDied","Data":"adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c"} Jul 07 14:11:11.970862 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.970769 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8aec824-073a-42ec-94b8-25c9e535cbdf-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-zzxlf\" (UID: \"f8aec824-073a-42ec-94b8-25c9e535cbdf\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf" Jul 07 14:11:11.970862 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:11.970822 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqnzs\" (UniqueName: \"kubernetes.io/projected/f8aec824-073a-42ec-94b8-25c9e535cbdf-kube-api-access-mqnzs\") pod \"monitoring-plugin-d5dcff7cf-zzxlf\" (UID: \"f8aec824-073a-42ec-94b8-25c9e535cbdf\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf" Jul 07 14:11:12.001229 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:12.001195 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:12.072501 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:12.072459 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8aec824-073a-42ec-94b8-25c9e535cbdf-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-zzxlf\" (UID: \"f8aec824-073a-42ec-94b8-25c9e535cbdf\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf" Jul 07 14:11:12.072684 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:12.072620 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqnzs\" (UniqueName: \"kubernetes.io/projected/f8aec824-073a-42ec-94b8-25c9e535cbdf-kube-api-access-mqnzs\") pod \"monitoring-plugin-d5dcff7cf-zzxlf\" (UID: \"f8aec824-073a-42ec-94b8-25c9e535cbdf\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf" Jul 07 14:11:12.075011 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:12.074981 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f8aec824-073a-42ec-94b8-25c9e535cbdf-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-zzxlf\" (UID: \"f8aec824-073a-42ec-94b8-25c9e535cbdf\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf" Jul 07 14:11:12.081222 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:12.081202 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqnzs\" (UniqueName: \"kubernetes.io/projected/f8aec824-073a-42ec-94b8-25c9e535cbdf-kube-api-access-mqnzs\") pod \"monitoring-plugin-d5dcff7cf-zzxlf\" (UID: \"f8aec824-073a-42ec-94b8-25c9e535cbdf\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf" Jul 07 14:11:12.129571 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:12.129545 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf" Jul 07 14:11:12.257586 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:12.257495 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf"] Jul 07 14:11:12.260005 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:11:12.259982 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8aec824_073a_42ec_94b8_25c9e535cbdf.slice/crio-51d9ddf86759660ac0771891ba09f4c6d611f3fb23058254044514af26b82c6e WatchSource:0}: Error finding container 51d9ddf86759660ac0771891ba09f4c6d611f3fb23058254044514af26b82c6e: Status 404 returned error can't find the container with id 51d9ddf86759660ac0771891ba09f4c6d611f3fb23058254044514af26b82c6e Jul 07 14:11:12.918415 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:12.918378 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf" event={"ID":"f8aec824-073a-42ec-94b8-25c9e535cbdf","Type":"ContainerStarted","Data":"51d9ddf86759660ac0771891ba09f4c6d611f3fb23058254044514af26b82c6e"} Jul 07 14:11:13.256288 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:13.256187 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:14.927900 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:14.927857 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf" event={"ID":"f8aec824-073a-42ec-94b8-25c9e535cbdf","Type":"ContainerStarted","Data":"4f179f782dfd253e758de6492976cf7f276360db160967237b965edfb3ac3c98"} Jul 07 14:11:14.928380 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:14.928055 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf" Jul 07 14:11:14.929663 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:14.929631 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" event={"ID":"74877d22-3734-41b4-a37c-158de4eabac0","Type":"ContainerStarted","Data":"847a2ed989c9e924e0af594f8f4db9692fda2aff35bb31b9fc10560a5f5ccee8"} Jul 07 14:11:14.932978 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:14.932956 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c09135-a711-4e32-8327-84101b838752","Type":"ContainerStarted","Data":"a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79"} Jul 07 14:11:14.933077 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:14.933018 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c09135-a711-4e32-8327-84101b838752","Type":"ContainerStarted","Data":"c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8"} Jul 07 14:11:14.933077 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:14.933061 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c09135-a711-4e32-8327-84101b838752","Type":"ContainerStarted","Data":"a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32"} Jul 07 14:11:14.933194 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:14.933100 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c09135-a711-4e32-8327-84101b838752","Type":"ContainerStarted","Data":"6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac"} Jul 07 14:11:14.933194 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:14.933137 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c09135-a711-4e32-8327-84101b838752","Type":"ContainerStarted","Data":"6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a"} Jul 07 14:11:14.933397 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:14.933375 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf" Jul 07 14:11:14.943707 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:14.943665 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-zzxlf" podStartSLOduration=1.971260585 podStartE2EDuration="3.943653626s" totalImagesPullingTime="1.972393041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:14.943482665 +0000 UTC m=+233.375725170" watchObservedRunningTime="2026-07-07 14:11:14.943653626 +0000 UTC m=+233.375896106" Jul 07 14:11:14.960304 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:14.960204 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" podStartSLOduration=1.620772689 podStartE2EDuration="3.960185821s" totalImagesPullingTime="2.339413132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:14.959934691 +0000 UTC m=+233.392177196" watchObservedRunningTime="2026-07-07 14:11:14.960185821 +0000 UTC m=+233.392428327" Jul 07 14:11:14.978990 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:14.978953 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" podUID="98855958-0a9b-47ca-93a9-8b82df6c7ee6" containerName="registry" containerID="cri-o://f6430c9f5f71e388fad7b144189a0d331b95c9c0c7e7a53512c35dbd65746c47" gracePeriod=30 Jul 07 14:11:15.221767 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.221739 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:11:15.297561 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.297532 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98855958-0a9b-47ca-93a9-8b82df6c7ee6-ca-trust-extracted\") pod \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " Jul 07 14:11:15.297726 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.297575 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98855958-0a9b-47ca-93a9-8b82df6c7ee6-installation-pull-secrets\") pod \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " Jul 07 14:11:15.297726 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.297634 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-certificates\") pod \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " Jul 07 14:11:15.297726 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.297665 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls\") pod \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " Jul 07 14:11:15.297726 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.297697 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgxkn\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-kube-api-access-pgxkn\") pod \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " Jul 07 14:11:15.297726 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.297721 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-bound-sa-token\") pod \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " Jul 07 14:11:15.298000 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.297750 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98855958-0a9b-47ca-93a9-8b82df6c7ee6-image-registry-private-configuration\") pod \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " Jul 07 14:11:15.298000 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.297787 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98855958-0a9b-47ca-93a9-8b82df6c7ee6-trusted-ca\") pod \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\" (UID: \"98855958-0a9b-47ca-93a9-8b82df6c7ee6\") " Jul 07 14:11:15.298571 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.298516 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98855958-0a9b-47ca-93a9-8b82df6c7ee6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "98855958-0a9b-47ca-93a9-8b82df6c7ee6" (UID: "98855958-0a9b-47ca-93a9-8b82df6c7ee6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:11:15.299150 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.299114 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "98855958-0a9b-47ca-93a9-8b82df6c7ee6" (UID: "98855958-0a9b-47ca-93a9-8b82df6c7ee6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:11:15.301590 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.301568 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98855958-0a9b-47ca-93a9-8b82df6c7ee6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "98855958-0a9b-47ca-93a9-8b82df6c7ee6" (UID: "98855958-0a9b-47ca-93a9-8b82df6c7ee6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:11:15.301691 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.301677 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "98855958-0a9b-47ca-93a9-8b82df6c7ee6" (UID: "98855958-0a9b-47ca-93a9-8b82df6c7ee6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:11:15.301750 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.301698 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98855958-0a9b-47ca-93a9-8b82df6c7ee6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "98855958-0a9b-47ca-93a9-8b82df6c7ee6" (UID: "98855958-0a9b-47ca-93a9-8b82df6c7ee6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:11:15.301816 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.301747 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "98855958-0a9b-47ca-93a9-8b82df6c7ee6" (UID: "98855958-0a9b-47ca-93a9-8b82df6c7ee6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:11:15.302059 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.302042 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-kube-api-access-pgxkn" (OuterVolumeSpecName: "kube-api-access-pgxkn") pod "98855958-0a9b-47ca-93a9-8b82df6c7ee6" (UID: "98855958-0a9b-47ca-93a9-8b82df6c7ee6"). InnerVolumeSpecName "kube-api-access-pgxkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:11:15.308569 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.308545 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98855958-0a9b-47ca-93a9-8b82df6c7ee6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "98855958-0a9b-47ca-93a9-8b82df6c7ee6" (UID: "98855958-0a9b-47ca-93a9-8b82df6c7ee6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:11:15.399336 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.399295 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-tls\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:11:15.399336 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.399326 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgxkn\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-kube-api-access-pgxkn\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:11:15.399336 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.399336 2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98855958-0a9b-47ca-93a9-8b82df6c7ee6-bound-sa-token\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:11:15.399336 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.399345 2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98855958-0a9b-47ca-93a9-8b82df6c7ee6-image-registry-private-configuration\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:11:15.399568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.399354 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98855958-0a9b-47ca-93a9-8b82df6c7ee6-trusted-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:11:15.399568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.399363 2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98855958-0a9b-47ca-93a9-8b82df6c7ee6-ca-trust-extracted\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:11:15.399568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.399371 2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98855958-0a9b-47ca-93a9-8b82df6c7ee6-installation-pull-secrets\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:11:15.399568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.399378 2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98855958-0a9b-47ca-93a9-8b82df6c7ee6-registry-certificates\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:11:15.937704 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.937676 2560 generic.go:356] "Generic (PLEG): container finished" podID="98855958-0a9b-47ca-93a9-8b82df6c7ee6" containerID="f6430c9f5f71e388fad7b144189a0d331b95c9c0c7e7a53512c35dbd65746c47" exitCode=0 Jul 07 14:11:15.938050 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.937718 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" Jul 07 14:11:15.938050 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.937749 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" event={"ID":"98855958-0a9b-47ca-93a9-8b82df6c7ee6","Type":"ContainerDied","Data":"f6430c9f5f71e388fad7b144189a0d331b95c9c0c7e7a53512c35dbd65746c47"} Jul 07 14:11:15.938050 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.937794 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68c89555f7-pxm8l" event={"ID":"98855958-0a9b-47ca-93a9-8b82df6c7ee6","Type":"ContainerDied","Data":"49e2b35830cf810573c828c392646820d86407c37dc7813c6499ba0f787eccc9"} Jul 07 14:11:15.938050 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.937824 2560 scope.go:117] "RemoveContainer" containerID="f6430c9f5f71e388fad7b144189a0d331b95c9c0c7e7a53512c35dbd65746c47" Jul 07 14:11:15.941308 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.941279 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c09135-a711-4e32-8327-84101b838752","Type":"ContainerStarted","Data":"d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d"} Jul 07 14:11:15.946373 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.946359 2560 scope.go:117] "RemoveContainer" containerID="f6430c9f5f71e388fad7b144189a0d331b95c9c0c7e7a53512c35dbd65746c47" Jul 07 14:11:15.946621 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:11:15.946602 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6430c9f5f71e388fad7b144189a0d331b95c9c0c7e7a53512c35dbd65746c47\": container with ID starting with f6430c9f5f71e388fad7b144189a0d331b95c9c0c7e7a53512c35dbd65746c47 not found: ID does not exist" containerID="f6430c9f5f71e388fad7b144189a0d331b95c9c0c7e7a53512c35dbd65746c47" Jul 07 14:11:15.946659 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.946631 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6430c9f5f71e388fad7b144189a0d331b95c9c0c7e7a53512c35dbd65746c47"} err="failed to get container status \"f6430c9f5f71e388fad7b144189a0d331b95c9c0c7e7a53512c35dbd65746c47\": rpc error: code = NotFound desc = could not find container \"f6430c9f5f71e388fad7b144189a0d331b95c9c0c7e7a53512c35dbd65746c47\": container with ID starting with f6430c9f5f71e388fad7b144189a0d331b95c9c0c7e7a53512c35dbd65746c47 not found: ID does not exist" Jul 07 14:11:15.971548 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.971502 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.499472428 podStartE2EDuration="7.971489358s" totalImagesPullingTime="4.455909124s" totalInitContainerRuntime="16.107806ms" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:15.970765997 +0000 UTC m=+234.403008499" watchObservedRunningTime="2026-07-07 14:11:15.971489358 +0000 UTC m=+234.403731858" Jul 07 14:11:15.997070 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:15.997049 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68c89555f7-pxm8l"] Jul 07 14:11:16.002528 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:16.002510 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68c89555f7-pxm8l"] Jul 07 14:11:16.211433 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:16.211353 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98855958-0a9b-47ca-93a9-8b82df6c7ee6" path="/var/lib/kubelet/pods/98855958-0a9b-47ca-93a9-8b82df6c7ee6/volumes" Jul 07 14:11:17.292433 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.292393 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-s7c7g"] Jul 07 14:11:17.292816 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.292686 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98855958-0a9b-47ca-93a9-8b82df6c7ee6" containerName="registry" Jul 07 14:11:17.292816 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.292698 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="98855958-0a9b-47ca-93a9-8b82df6c7ee6" containerName="registry" Jul 07 14:11:17.292816 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.292763 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="98855958-0a9b-47ca-93a9-8b82df6c7ee6" containerName="registry" Jul 07 14:11:17.297202 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.297185 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-s7c7g" Jul 07 14:11:17.299431 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.299406 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 14:11:17.299585 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.299569 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 14:11:17.299644 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.299626 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lzw7h\"" Jul 07 14:11:17.303013 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.302994 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:17.306819 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.306797 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-s7c7g"] Jul 07 14:11:17.316729 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.316709 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8znb\" (UniqueName: \"kubernetes.io/projected/93bad8d3-1a35-486c-b1e6-539ced4e3b45-kube-api-access-p8znb\") pod \"downloads-685bfb5f64-s7c7g\" (UID: \"93bad8d3-1a35-486c-b1e6-539ced4e3b45\") " pod="openshift-console/downloads-685bfb5f64-s7c7g" Jul 07 14:11:17.399578 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.399544 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:17.417416 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.417390 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8znb\" (UniqueName: \"kubernetes.io/projected/93bad8d3-1a35-486c-b1e6-539ced4e3b45-kube-api-access-p8znb\") pod \"downloads-685bfb5f64-s7c7g\" (UID: \"93bad8d3-1a35-486c-b1e6-539ced4e3b45\") " pod="openshift-console/downloads-685bfb5f64-s7c7g" Jul 07 14:11:17.425563 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.425541 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8znb\" (UniqueName: \"kubernetes.io/projected/93bad8d3-1a35-486c-b1e6-539ced4e3b45-kube-api-access-p8znb\") pod \"downloads-685bfb5f64-s7c7g\" (UID: \"93bad8d3-1a35-486c-b1e6-539ced4e3b45\") " pod="openshift-console/downloads-685bfb5f64-s7c7g" Jul 07 14:11:17.500035 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.500004 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:17.599337 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.599233 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:17.607014 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.606991 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-s7c7g" Jul 07 14:11:17.737651 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.737602 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-s7c7g"] Jul 07 14:11:17.739593 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:11:17.739566 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bad8d3_1a35_486c_b1e6_539ced4e3b45.slice/crio-acbb98e9354b735f7dc499cb5cc6bef8689b2235072cbe852da2edf8ae1a03f0 WatchSource:0}: Error finding container acbb98e9354b735f7dc499cb5cc6bef8689b2235072cbe852da2edf8ae1a03f0: Status 404 returned error can't find the container with id acbb98e9354b735f7dc499cb5cc6bef8689b2235072cbe852da2edf8ae1a03f0 Jul 07 14:11:17.952374 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.952296 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-s7c7g" event={"ID":"93bad8d3-1a35-486c-b1e6-539ced4e3b45","Type":"ContainerStarted","Data":"acbb98e9354b735f7dc499cb5cc6bef8689b2235072cbe852da2edf8ae1a03f0"} Jul 07 14:11:17.956515 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:17.956494 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:18.056282 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:18.056237 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:18.174043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:18.174010 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:18.277661 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:18.277592 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:19.165953 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:19.165921 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:19.365190 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:19.365162 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:19.763928 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:19.763885 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:19.987404 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:19.987368 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:20.085679 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:20.085624 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:20.316356 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:20.316316 2560 patch_prober.go:28] interesting pod/image-registry-7d85bcc555-qb8tv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 14:11:20.316777 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:20.316376 2560 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" podUID="ff734b15-88fd-402d-8c78-0a60cb48c5d8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:11:21.834172 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:21.834137 2560 patch_prober.go:28] interesting pod/image-registry-7d85bcc555-qb8tv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 14:11:21.834707 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:21.834204 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" podUID="ff734b15-88fd-402d-8c78-0a60cb48c5d8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:11:27.123060 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.123024 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:27.198173 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.198143 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:27.202026 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.202001 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d5d56c99b-2xj7k"] Jul 07 14:11:27.207358 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.207336 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.210916 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.210822 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 14:11:27.210916 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.210877 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 14:11:27.210916 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.210891 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 14:11:27.211141 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.210878 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 14:11:27.211141 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.210890 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8fh2z\"" Jul 07 14:11:27.211141 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.210876 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 14:11:27.216709 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.216690 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5d56c99b-2xj7k"] Jul 07 14:11:27.306581 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.306550 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-oauth-config\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.306752 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.306676 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-oauth-serving-cert\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.306752 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.306703 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xfdq\" (UniqueName: \"kubernetes.io/projected/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-kube-api-access-6xfdq\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.306752 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.306735 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-serving-cert\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.306916 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.306762 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-service-ca\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.306916 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.306862 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-config\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.407531 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.407447 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-serving-cert\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.407531 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.407491 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-service-ca\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.407531 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.407524 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-config\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.407795 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.407580 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-oauth-config\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.407795 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.407679 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-oauth-serving-cert\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.407795 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.407703 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xfdq\" (UniqueName: \"kubernetes.io/projected/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-kube-api-access-6xfdq\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.408759 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.408732 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-config\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.408895 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.408773 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-service-ca\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.408895 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.408825 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-oauth-serving-cert\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.410359 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.410335 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-oauth-config\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.410584 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.410561 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-serving-cert\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.417984 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.417957 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xfdq\" (UniqueName: \"kubernetes.io/projected/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-kube-api-access-6xfdq\") pod \"console-5d5d56c99b-2xj7k\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.518688 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.518646 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:27.661976 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.661907 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5d56c99b-2xj7k"] Jul 07 14:11:27.665149 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:11:27.665126 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9157a9_eecc_4fcc_84f6_47f1bcc915d7.slice/crio-f743f2c909dc5af6f836d004682aa59139cf37a52a13494ccff7b55ace217795 WatchSource:0}: Error finding container f743f2c909dc5af6f836d004682aa59139cf37a52a13494ccff7b55ace217795: Status 404 returned error can't find the container with id f743f2c909dc5af6f836d004682aa59139cf37a52a13494ccff7b55ace217795 Jul 07 14:11:27.993899 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:27.993817 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5d56c99b-2xj7k" event={"ID":"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7","Type":"ContainerStarted","Data":"f743f2c909dc5af6f836d004682aa59139cf37a52a13494ccff7b55ace217795"} Jul 07 14:11:29.700485 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:29.700392 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:29.773649 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:29.773603 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:30.318376 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:30.318322 2560 patch_prober.go:28] interesting pod/image-registry-7d85bcc555-qb8tv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 14:11:30.318566 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:30.318402 2560 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" podUID="ff734b15-88fd-402d-8c78-0a60cb48c5d8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:11:30.318566 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:30.318446 2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:11:30.319043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:30.319014 2560 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"ae679b40196ca588eaf0e30bda704837a1326b686264a3c7f0c3c12828dce0ef"} pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" containerMessage="Container registry failed liveness probe, will be restarted" Jul 07 14:11:30.323587 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:30.323557 2560 patch_prober.go:28] interesting pod/image-registry-7d85bcc555-qb8tv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 14:11:30.323729 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:30.323602 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" podUID="ff734b15-88fd-402d-8c78-0a60cb48c5d8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:11:30.788668 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:30.788590 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:30.887939 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:30.887907 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:31.758308 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:31.758255 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:31.758479 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:31.758340 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:32.193206 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:32.193173 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:32.254180 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:32.254147 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:33.970098 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:33.970060 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs\") pod \"network-metrics-daemon-8rf2w\" (UID: \"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603\") " pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:11:33.972759 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:33.972730 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d5ba7ef-4b44-4c07-9dc0-77a3e4223603-metrics-certs\") pod \"network-metrics-daemon-8rf2w\" (UID: \"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603\") " pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:11:34.007445 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:34.007411 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wgdzp\"" Jul 07 14:11:34.015564 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:34.015537 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8rf2w" Jul 07 14:11:34.971764 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:34.971717 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:35.099958 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.099923 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:35.778783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.778749 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65d5f7444-pl479"] Jul 07 14:11:35.792847 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.792816 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65d5f7444-pl479"] Jul 07 14:11:35.793030 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.792941 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.800946 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.800918 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 14:11:35.895088 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.895061 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/944599fc-10ae-4a3a-850a-8206bb28f61b-console-oauth-config\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.895285 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.895095 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-trusted-ca-bundle\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.895285 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.895159 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-console-config\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.895402 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.895286 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/944599fc-10ae-4a3a-850a-8206bb28f61b-console-serving-cert\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.895402 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.895331 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hks5t\" (UniqueName: \"kubernetes.io/projected/944599fc-10ae-4a3a-850a-8206bb28f61b-kube-api-access-hks5t\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.895402 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.895366 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-oauth-serving-cert\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.895516 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.895406 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-service-ca\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.981639 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.981605 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:35.996893 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.996857 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/944599fc-10ae-4a3a-850a-8206bb28f61b-console-serving-cert\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.997045 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.996905 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hks5t\" (UniqueName: \"kubernetes.io/projected/944599fc-10ae-4a3a-850a-8206bb28f61b-kube-api-access-hks5t\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.997045 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.996949 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-oauth-serving-cert\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.997045 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.996993 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-service-ca\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.997045 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.997022 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/944599fc-10ae-4a3a-850a-8206bb28f61b-console-oauth-config\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.997367 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.997053 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-trusted-ca-bundle\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.997367 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.997092 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-console-config\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.998131 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.998103 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-service-ca\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.998228 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.998152 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-console-config\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.998228 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.998156 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-trusted-ca-bundle\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.998228 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.998201 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-oauth-serving-cert\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.999820 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.999795 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/944599fc-10ae-4a3a-850a-8206bb28f61b-console-serving-cert\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:35.999975 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:35.999950 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/944599fc-10ae-4a3a-850a-8206bb28f61b-console-oauth-config\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:36.005900 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:36.005881 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hks5t\" (UniqueName: \"kubernetes.io/projected/944599fc-10ae-4a3a-850a-8206bb28f61b-kube-api-access-hks5t\") pod \"console-65d5f7444-pl479\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:36.105044 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:36.105006 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:37.280135 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:37.279938 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65d5f7444-pl479"] Jul 07 14:11:37.282108 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:11:37.282075 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944599fc_10ae_4a3a_850a_8206bb28f61b.slice/crio-d94cc1cd98c77055d3ed201d915f1703e4a1b7a1dfbd8215f07fca3dd253dee3 WatchSource:0}: Error finding container d94cc1cd98c77055d3ed201d915f1703e4a1b7a1dfbd8215f07fca3dd253dee3: Status 404 returned error can't find the container with id d94cc1cd98c77055d3ed201d915f1703e4a1b7a1dfbd8215f07fca3dd253dee3 Jul 07 14:11:37.294681 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:37.294657 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8rf2w"] Jul 07 14:11:37.299482 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:11:37.299460 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5ba7ef_4b44_4c07_9dc0_77a3e4223603.slice/crio-c69d3fc1c8ba1fb01a529f935b97aff0ba56eff2825251283a5da10c8ceb8ceb WatchSource:0}: Error finding container c69d3fc1c8ba1fb01a529f935b97aff0ba56eff2825251283a5da10c8ceb8ceb: Status 404 returned error can't find the container with id c69d3fc1c8ba1fb01a529f935b97aff0ba56eff2825251283a5da10c8ceb8ceb Jul 07 14:11:38.039826 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:38.039774 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5d56c99b-2xj7k" event={"ID":"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7","Type":"ContainerStarted","Data":"4ea8580bdbca714d181472c35e6232759f1c86bce0f354065503bbcb10ac6749"} Jul 07 14:11:38.041963 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:38.041881 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d5f7444-pl479" event={"ID":"944599fc-10ae-4a3a-850a-8206bb28f61b","Type":"ContainerStarted","Data":"53453a2cbea1c7ef54051c4d8e407703363cdde7cecc61466e17038781aa2792"} Jul 07 14:11:38.041963 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:38.041931 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d5f7444-pl479" event={"ID":"944599fc-10ae-4a3a-850a-8206bb28f61b","Type":"ContainerStarted","Data":"d94cc1cd98c77055d3ed201d915f1703e4a1b7a1dfbd8215f07fca3dd253dee3"} Jul 07 14:11:38.045066 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:38.045039 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8rf2w" event={"ID":"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603","Type":"ContainerStarted","Data":"c69d3fc1c8ba1fb01a529f935b97aff0ba56eff2825251283a5da10c8ceb8ceb"} Jul 07 14:11:38.047202 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:38.047170 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-s7c7g" event={"ID":"93bad8d3-1a35-486c-b1e6-539ced4e3b45","Type":"ContainerStarted","Data":"ebfb7879b57babdbc0d9c973eb038951ec98d71bffcab22abc7290787e40efba"} Jul 07 14:11:38.047454 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:38.047434 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-s7c7g" Jul 07 14:11:38.058930 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:38.058890 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d5d56c99b-2xj7k" podStartSLOduration=1.585779911 podStartE2EDuration="11.058877362s" totalImagesPullingTime="9.473097451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:38.056565395 +0000 UTC m=+256.488807899" watchObservedRunningTime="2026-07-07 14:11:38.058877362 +0000 UTC m=+256.491119863" Jul 07 14:11:38.059825 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:38.059795 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-s7c7g" Jul 07 14:11:38.090844 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:38.090215 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-s7c7g" podStartSLOduration=1.6476348650000001 podStartE2EDuration="21.090199994s" totalImagesPullingTime="19.442565129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:38.089525179 +0000 UTC m=+256.521767682" watchObservedRunningTime="2026-07-07 14:11:38.090199994 +0000 UTC m=+256.522442494" Jul 07 14:11:38.091027 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:38.090995 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65d5f7444-pl479" podStartSLOduration=3.090982585 podStartE2EDuration="3.090982585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:38.07548103 +0000 UTC m=+256.507723551" watchObservedRunningTime="2026-07-07 14:11:38.090982585 +0000 UTC m=+256.523225087" Jul 07 14:11:39.055621 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:39.054988 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8rf2w" event={"ID":"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603","Type":"ContainerStarted","Data":"813edcf3a983c8e262a3367d0b733807854afa009dea7b3113504942f5ddb924"} Jul 07 14:11:39.055621 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:39.055098 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8rf2w" event={"ID":"3d5ba7ef-4b44-4c07-9dc0-77a3e4223603","Type":"ContainerStarted","Data":"e6b8cedac86940b349ac5bc5c0d4518840cc217f4052f1a4c95edf2b1ff7d7bd"} Jul 07 14:11:39.072389 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:39.072324 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8rf2w" podStartSLOduration=254.918582873 podStartE2EDuration="4m16.072303729s" totalImagesPullingTime="1.153720856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:11:39.070984845 +0000 UTC m=+257.503227347" watchObservedRunningTime="2026-07-07 14:11:39.072303729 +0000 UTC m=+257.504546232" Jul 07 14:11:39.167992 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:39.167952 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:39.266382 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:39.266334 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:40.323547 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:40.323512 2560 patch_prober.go:28] interesting pod/image-registry-7d85bcc555-qb8tv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 14:11:40.323951 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:40.323572 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" podUID="ff734b15-88fd-402d-8c78-0a60cb48c5d8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:11:43.787203 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:43.787117 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:43.847018 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:43.846962 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:44.838204 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:44.838156 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:44.938817 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:44.938780 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:46.105469 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:46.105424 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:46.105927 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:46.105495 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:46.107216 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:46.107178 2560 patch_prober.go:28] interesting pod/console-65d5f7444-pl479 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.21:8443/health\": dial tcp 10.134.0.21:8443: connect: connection refused" start-of-body= Jul 07 14:11:46.107361 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:46.107230 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-65d5f7444-pl479" podUID="944599fc-10ae-4a3a-850a-8206bb28f61b" containerName="console" probeResult="failure" output="Get \"https://10.134.0.21:8443/health\": dial tcp 10.134.0.21:8443: connect: connection refused" Jul 07 14:11:47.519824 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:47.519791 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:47.520237 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:47.519837 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:11:47.521314 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:47.521291 2560 patch_prober.go:28] interesting pod/console-5d5d56c99b-2xj7k container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.20:8443/health\": dial tcp 10.134.0.20:8443: connect: connection refused" start-of-body= Jul 07 14:11:47.521426 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:47.521330 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5d5d56c99b-2xj7k" podUID="cc9157a9-eecc-4fcc-84f6-47f1bcc915d7" containerName="console" probeResult="failure" output="Get \"https://10.134.0.20:8443/health\": dial tcp 10.134.0.20:8443: connect: connection refused" Jul 07 14:11:49.095779 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:49.095741 2560 generic.go:356] "Generic (PLEG): container finished" podID="b9a6eef4-26d4-4d5c-a1ea-6d71968d1116" containerID="db0c78e66027eff53ae3034cbe6d076cb74ceb725bc0370df316be4f632e8566" exitCode=0 Jul 07 14:11:49.096174 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:49.095812 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" event={"ID":"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116","Type":"ContainerDied","Data":"db0c78e66027eff53ae3034cbe6d076cb74ceb725bc0370df316be4f632e8566"} Jul 07 14:11:49.096174 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:49.096137 2560 scope.go:117] "RemoveContainer" containerID="db0c78e66027eff53ae3034cbe6d076cb74ceb725bc0370df316be4f632e8566" Jul 07 14:11:50.101375 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:50.101341 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-hwj87" event={"ID":"b9a6eef4-26d4-4d5c-a1ea-6d71968d1116","Type":"ContainerStarted","Data":"90b3610c76b785893703f9e9ad443fbf1895f3c8a092ce2c51e8df81b6eea38a"} Jul 07 14:11:50.322936 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:50.322903 2560 patch_prober.go:28] interesting pod/image-registry-7d85bcc555-qb8tv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 14:11:50.323122 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:50.322964 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" podUID="ff734b15-88fd-402d-8c78-0a60cb48c5d8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 14:11:51.763534 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:51.763503 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:51.767154 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:51.767132 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-f6c6c6887-mgxfb" Jul 07 14:11:52.116131 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:52.116106 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f5c09135-a711-4e32-8327-84101b838752/init-config-reloader/0.log" Jul 07 14:11:52.316389 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:52.316357 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f5c09135-a711-4e32-8327-84101b838752/alertmanager/0.log" Jul 07 14:11:52.519217 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:52.519145 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f5c09135-a711-4e32-8327-84101b838752/config-reloader/0.log" Jul 07 14:11:52.716300 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:52.716256 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f5c09135-a711-4e32-8327-84101b838752/kube-rbac-proxy-web/0.log" Jul 07 14:11:52.917000 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:52.916975 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f5c09135-a711-4e32-8327-84101b838752/kube-rbac-proxy/0.log" Jul 07 14:11:53.116240 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:53.116202 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f5c09135-a711-4e32-8327-84101b838752/kube-rbac-proxy-metric/0.log" Jul 07 14:11:53.316127 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:53.316096 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f5c09135-a711-4e32-8327-84101b838752/prom-label-proxy/0.log" Jul 07 14:11:53.517657 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:53.517629 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-w76w5_fd5fab7c-5de3-4f92-85a9-863fd04af85d/cluster-monitoring-operator/0.log" Jul 07 14:11:54.316689 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:54.316654 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-f6c6c6887-mgxfb_74877d22-3734-41b4-a37c-158de4eabac0/metrics-server/0.log" Jul 07 14:11:54.495774 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:54.495724 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:54.516290 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:54.516253 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-zzxlf_f8aec824-073a-42ec-94b8-25c9e535cbdf/monitoring-plugin/0.log" Jul 07 14:11:55.342036 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:55.341997 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" podUID="ff734b15-88fd-402d-8c78-0a60cb48c5d8" containerName="registry" containerID="cri-o://ae679b40196ca588eaf0e30bda704837a1326b686264a3c7f0c3c12828dce0ef" gracePeriod=30 Jul 07 14:11:55.917374 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:55.917344 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nz8cf_de614e33-c9bd-4ba8-850f-8a49fc2b0607/init-textfile/0.log" Jul 07 14:11:56.109686 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:56.109660 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:56.113628 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:56.113607 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:11:56.116673 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:56.116658 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nz8cf_de614e33-c9bd-4ba8-850f-8a49fc2b0607/node-exporter/0.log" Jul 07 14:11:56.169957 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:56.169887 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d5d56c99b-2xj7k"] Jul 07 14:11:56.316299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:56.316251 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nz8cf_de614e33-c9bd-4ba8-850f-8a49fc2b0607/kube-rbac-proxy/0.log" Jul 07 14:11:56.941362 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:56.941328 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:11:57.129300 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:57.129244 2560 generic.go:356] "Generic (PLEG): container finished" podID="ff734b15-88fd-402d-8c78-0a60cb48c5d8" containerID="ae679b40196ca588eaf0e30bda704837a1326b686264a3c7f0c3c12828dce0ef" exitCode=0 Jul 07 14:11:57.129484 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:57.129306 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" event={"ID":"ff734b15-88fd-402d-8c78-0a60cb48c5d8","Type":"ContainerDied","Data":"ae679b40196ca588eaf0e30bda704837a1326b686264a3c7f0c3c12828dce0ef"} Jul 07 14:11:57.129484 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:57.129344 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" event={"ID":"ff734b15-88fd-402d-8c78-0a60cb48c5d8","Type":"ContainerStarted","Data":"fd8154eb18d959a83b80df8ef144f7b9e46f657e857d4e4fdc54588d373dcbbf"} Jul 07 14:11:57.129484 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:57.129442 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:11:58.134180 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:58.134143 2560 generic.go:356] "Generic (PLEG): container finished" podID="c520cb66-9856-40eb-981c-ad7a9687928a" containerID="33b9d9730daf063e4f9ad566841d85dac8c20a1f7f7372b0dce5a7b31760d48a" exitCode=0 Jul 07 14:11:58.134762 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:58.134223 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" event={"ID":"c520cb66-9856-40eb-981c-ad7a9687928a","Type":"ContainerDied","Data":"33b9d9730daf063e4f9ad566841d85dac8c20a1f7f7372b0dce5a7b31760d48a"} Jul 07 14:11:58.134762 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:58.134711 2560 scope.go:117] "RemoveContainer" containerID="33b9d9730daf063e4f9ad566841d85dac8c20a1f7f7372b0dce5a7b31760d48a" Jul 07 14:11:59.139575 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:11:59.139543 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-nrjbq" event={"ID":"c520cb66-9856-40eb-981c-ad7a9687928a","Type":"ContainerStarted","Data":"c752d16c5a91d801c85dbb1abd75c1051033d4810caac4467d3f089c853385bf"} Jul 07 14:12:01.201200 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:01.201166 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:01.256145 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:01.256111 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:01.384926 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:01.384894 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:01.434210 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:01.434183 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:01.517195 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:01.517124 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d5d56c99b-2xj7k_cc9157a9-eecc-4fcc-84f6-47f1bcc915d7/console/0.log" Jul 07 14:12:01.715951 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:01.715917 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65d5f7444-pl479_944599fc-10ae-4a3a-850a-8206bb28f61b/console/0.log" Jul 07 14:12:01.917240 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:01.917212 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-s7c7g_93bad8d3-1a35-486c-b1e6-539ced4e3b45/download-server/0.log" Jul 07 14:12:03.517033 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:03.517011 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6bc6n_1ac773f2-d330-4140-97aa-8ac19ee4d555/dns-node-resolver/0.log" Jul 07 14:12:03.682440 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:12:03.682402 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hkczl" podUID="b88eafd6-43c8-460b-b450-7b2714fb1a9c" Jul 07 14:12:04.116873 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:04.116848 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d85bcc555-qb8tv_ff734b15-88fd-402d-8c78-0a60cb48c5d8/registry/1.log" Jul 07 14:12:04.158952 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:04.158922 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hkczl" Jul 07 14:12:04.317235 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:04.317203 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5kt5s_1053d78f-a5ae-4acd-9a99-32beb279bbb4/node-ca/0.log" Jul 07 14:12:04.969986 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:04.969945 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:05.070848 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:05.070820 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:05.116725 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:05.116697 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5cgb6_a334bf52-3bf3-4862-8680-d4e89f3d28bc/serve-healthcheck-canary/0.log" Jul 07 14:12:05.716556 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:05.716521 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:05.774752 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:05.774720 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:07.388224 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:07.388180 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:12:07.390875 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:07.390851 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b88eafd6-43c8-460b-b450-7b2714fb1a9c-metrics-tls\") pod \"dns-default-hkczl\" (UID: \"b88eafd6-43c8-460b-b450-7b2714fb1a9c\") " pod="openshift-dns/dns-default-hkczl" Jul 07 14:12:07.462457 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:07.462424 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cxvss\"" Jul 07 14:12:07.470886 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:07.470620 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hkczl" Jul 07 14:12:07.655832 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:07.655753 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hkczl"] Jul 07 14:12:07.659662 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:12:07.659636 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88eafd6_43c8_460b_b450_7b2714fb1a9c.slice/crio-8e2bf30fc1ea3524cd0b4c3708999c8f5e336f6168f35f61ffe323111b586442 WatchSource:0}: Error finding container 8e2bf30fc1ea3524cd0b4c3708999c8f5e336f6168f35f61ffe323111b586442: Status 404 returned error can't find the container with id 8e2bf30fc1ea3524cd0b4c3708999c8f5e336f6168f35f61ffe323111b586442 Jul 07 14:12:08.179208 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:08.179169 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hkczl" event={"ID":"b88eafd6-43c8-460b-b450-7b2714fb1a9c","Type":"ContainerStarted","Data":"8e2bf30fc1ea3524cd0b4c3708999c8f5e336f6168f35f61ffe323111b586442"} Jul 07 14:12:11.198107 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:11.198076 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hkczl" event={"ID":"b88eafd6-43c8-460b-b450-7b2714fb1a9c","Type":"ContainerStarted","Data":"5efd3ffbe3ea41dceca2fe98e91b7570d4b1e82c4dc036a95e236357c98a1c5d"} Jul 07 14:12:11.198581 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:11.198133 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hkczl" event={"ID":"b88eafd6-43c8-460b-b450-7b2714fb1a9c","Type":"ContainerStarted","Data":"d5045fb5abc3a856d30c3aac9f85365ad78d3b6f861974e6341651047c0d2119"} Jul 07 14:12:11.198581 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:11.198187 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hkczl" Jul 07 14:12:11.217863 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:11.217822 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hkczl" podStartSLOduration=251.47392735 podStartE2EDuration="4m14.217806998s" totalImagesPullingTime="2.743879648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:07:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:12:11.216102896 +0000 UTC m=+289.648345421" watchObservedRunningTime="2026-07-07 14:12:11.217806998 +0000 UTC m=+289.650049500" Jul 07 14:12:11.440453 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:11.440423 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:11.543695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:11.543668 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:11.644815 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:11.644781 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:14.803486 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:14.803455 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:14.863635 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:14.863601 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:18.139257 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:18.139225 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d85bcc555-qb8tv" Jul 07 14:12:18.270411 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:18.270380 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:18.369576 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:18.369544 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:19.991740 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:19.991707 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:20.051423 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:20.051391 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:21.190475 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.190432 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5d5d56c99b-2xj7k" podUID="cc9157a9-eecc-4fcc-84f6-47f1bcc915d7" containerName="console" containerID="cri-o://4ea8580bdbca714d181472c35e6232759f1c86bce0f354065503bbcb10ac6749" gracePeriod=15 Jul 07 14:12:21.203303 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.203280 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hkczl" Jul 07 14:12:21.322299 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.322257 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:21.381477 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.381446 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:21.486252 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.486224 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:12:21.616669 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.616638 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-serving-cert\") pod \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " Jul 07 14:12:21.616837 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.616677 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-oauth-config\") pod \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " Jul 07 14:12:21.616837 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.616729 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-service-ca\") pod \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " Jul 07 14:12:21.616837 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.616767 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-config\") pod \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " Jul 07 14:12:21.616837 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.616796 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xfdq\" (UniqueName: \"kubernetes.io/projected/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-kube-api-access-6xfdq\") pod \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " Jul 07 14:12:21.617038 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.616856 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-oauth-serving-cert\") pod \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\" (UID: \"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7\") " Jul 07 14:12:21.617215 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.617170 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-config" (OuterVolumeSpecName: "console-config") pod "cc9157a9-eecc-4fcc-84f6-47f1bcc915d7" (UID: "cc9157a9-eecc-4fcc-84f6-47f1bcc915d7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:12:21.617215 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.617182 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-service-ca" (OuterVolumeSpecName: "service-ca") pod "cc9157a9-eecc-4fcc-84f6-47f1bcc915d7" (UID: "cc9157a9-eecc-4fcc-84f6-47f1bcc915d7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:12:21.617215 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.617204 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cc9157a9-eecc-4fcc-84f6-47f1bcc915d7" (UID: "cc9157a9-eecc-4fcc-84f6-47f1bcc915d7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:12:21.619104 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.619079 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cc9157a9-eecc-4fcc-84f6-47f1bcc915d7" (UID: "cc9157a9-eecc-4fcc-84f6-47f1bcc915d7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:12:21.619174 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.619099 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cc9157a9-eecc-4fcc-84f6-47f1bcc915d7" (UID: "cc9157a9-eecc-4fcc-84f6-47f1bcc915d7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:12:21.619214 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.619195 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-kube-api-access-6xfdq" (OuterVolumeSpecName: "kube-api-access-6xfdq") pod "cc9157a9-eecc-4fcc-84f6-47f1bcc915d7" (UID: "cc9157a9-eecc-4fcc-84f6-47f1bcc915d7"). InnerVolumeSpecName "kube-api-access-6xfdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:12:21.718370 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.718290 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-config\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:21.718370 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.718321 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xfdq\" (UniqueName: \"kubernetes.io/projected/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-kube-api-access-6xfdq\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:21.718370 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.718335 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-oauth-serving-cert\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:21.718370 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.718346 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-serving-cert\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:21.718370 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.718359 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-console-oauth-config\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:21.718370 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:21.718371 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7-service-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:22.015468 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:22.015400 2560 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 14:12:22.355481 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:22.242358 2560 generic.go:356] "Generic (PLEG): container finished" podID="cc9157a9-eecc-4fcc-84f6-47f1bcc915d7" containerID="4ea8580bdbca714d181472c35e6232759f1c86bce0f354065503bbcb10ac6749" exitCode=0 Jul 07 14:12:22.355481 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:22.242406 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5d56c99b-2xj7k" event={"ID":"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7","Type":"ContainerDied","Data":"4ea8580bdbca714d181472c35e6232759f1c86bce0f354065503bbcb10ac6749"} Jul 07 14:12:22.355481 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:22.242416 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5d56c99b-2xj7k" Jul 07 14:12:22.355481 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:22.242449 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5d56c99b-2xj7k" event={"ID":"cc9157a9-eecc-4fcc-84f6-47f1bcc915d7","Type":"ContainerDied","Data":"f743f2c909dc5af6f836d004682aa59139cf37a52a13494ccff7b55ace217795"} Jul 07 14:12:22.355481 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:22.242470 2560 scope.go:117] "RemoveContainer" containerID="4ea8580bdbca714d181472c35e6232759f1c86bce0f354065503bbcb10ac6749" Jul 07 14:12:22.355481 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:22.250845 2560 scope.go:117] "RemoveContainer" containerID="4ea8580bdbca714d181472c35e6232759f1c86bce0f354065503bbcb10ac6749" Jul 07 14:12:22.355481 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:12:22.251077 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea8580bdbca714d181472c35e6232759f1c86bce0f354065503bbcb10ac6749\": container with ID starting with 4ea8580bdbca714d181472c35e6232759f1c86bce0f354065503bbcb10ac6749 not found: ID does not exist" containerID="4ea8580bdbca714d181472c35e6232759f1c86bce0f354065503bbcb10ac6749" Jul 07 14:12:22.355481 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:22.251100 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea8580bdbca714d181472c35e6232759f1c86bce0f354065503bbcb10ac6749"} err="failed to get container status \"4ea8580bdbca714d181472c35e6232759f1c86bce0f354065503bbcb10ac6749\": rpc error: code = NotFound desc = could not find container \"4ea8580bdbca714d181472c35e6232759f1c86bce0f354065503bbcb10ac6749\": container with ID starting with 4ea8580bdbca714d181472c35e6232759f1c86bce0f354065503bbcb10ac6749 not found: ID does not exist" Jul 07 14:12:22.355481 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:22.259697 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d5d56c99b-2xj7k"] Jul 07 14:12:22.355481 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:22.267309 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d5d56c99b-2xj7k"] Jul 07 14:12:23.147488 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:23.147453 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:23.247589 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:23.247557 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:24.207568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:24.207539 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc9157a9-eecc-4fcc-84f6-47f1bcc915d7" path="/var/lib/kubelet/pods/cc9157a9-eecc-4fcc-84f6-47f1bcc915d7/volumes" Jul 07 14:12:27.397307 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.397258 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:12:27.397690 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.397555 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="alertmanager" containerID="cri-o://6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a" gracePeriod=120 Jul 07 14:12:27.397690 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.397621 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="kube-rbac-proxy-metric" containerID="cri-o://a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79" gracePeriod=120 Jul 07 14:12:27.397808 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.397665 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="prom-label-proxy" containerID="cri-o://d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d" gracePeriod=120 Jul 07 14:12:27.397808 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.397686 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="kube-rbac-proxy-web" containerID="cri-o://a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32" gracePeriod=120 Jul 07 14:12:27.397808 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.397724 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="kube-rbac-proxy" containerID="cri-o://c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8" gracePeriod=120 Jul 07 14:12:27.397808 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.397751 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="config-reloader" containerID="cri-o://6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac" gracePeriod=120 Jul 07 14:12:27.449725 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.449693 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67554b6fdf-zd82b"] Jul 07 14:12:27.449978 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.449966 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9157a9-eecc-4fcc-84f6-47f1bcc915d7" containerName="console" Jul 07 14:12:27.450029 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.449980 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9157a9-eecc-4fcc-84f6-47f1bcc915d7" containerName="console" Jul 07 14:12:27.450068 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.450058 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9157a9-eecc-4fcc-84f6-47f1bcc915d7" containerName="console" Jul 07 14:12:27.452286 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.452241 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.463852 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.463827 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67554b6fdf-zd82b"] Jul 07 14:12:27.499472 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.499438 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:27.555991 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.555958 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:27.567189 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.567167 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vncz6\" (UniqueName: \"kubernetes.io/projected/c35722dd-f496-44ca-b846-46687b0f15dd-kube-api-access-vncz6\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.567298 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.567207 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c35722dd-f496-44ca-b846-46687b0f15dd-console-oauth-config\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.567298 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.567244 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-trusted-ca-bundle\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.567370 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.567307 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c35722dd-f496-44ca-b846-46687b0f15dd-console-serving-cert\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.567370 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.567334 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-service-ca\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.567370 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.567350 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-oauth-serving-cert\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.567458 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.567410 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-console-config\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.632810 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.632779 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:27.668026 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.667973 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c35722dd-f496-44ca-b846-46687b0f15dd-console-serving-cert\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.668026 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.668004 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-service-ca\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.668026 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.668019 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-oauth-serving-cert\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.668187 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.668064 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-console-config\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.668187 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.668091 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncz6\" (UniqueName: \"kubernetes.io/projected/c35722dd-f496-44ca-b846-46687b0f15dd-kube-api-access-vncz6\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.668187 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.668136 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c35722dd-f496-44ca-b846-46687b0f15dd-console-oauth-config\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.668357 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.668252 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-trusted-ca-bundle\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.668769 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.668743 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-service-ca\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.668916 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.668896 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-oauth-serving-cert\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.669117 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.669097 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-console-config\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.669167 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.669143 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-trusted-ca-bundle\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.670698 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.670677 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c35722dd-f496-44ca-b846-46687b0f15dd-console-serving-cert\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.670794 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.670737 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c35722dd-f496-44ca-b846-46687b0f15dd-console-oauth-config\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.675793 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.675771 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vncz6\" (UniqueName: \"kubernetes.io/projected/c35722dd-f496-44ca-b846-46687b0f15dd-kube-api-access-vncz6\") pod \"console-67554b6fdf-zd82b\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.762253 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.762227 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:27.894218 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:27.894191 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67554b6fdf-zd82b"] Jul 07 14:12:27.896569 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:12:27.896545 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35722dd_f496_44ca_b846_46687b0f15dd.slice/crio-ae9974052c1108bf559f75c1534bea8440a0da2d829f81356f587bba832fbfbb WatchSource:0}: Error finding container ae9974052c1108bf559f75c1534bea8440a0da2d829f81356f587bba832fbfbb: Status 404 returned error can't find the container with id ae9974052c1108bf559f75c1534bea8440a0da2d829f81356f587bba832fbfbb Jul 07 14:12:28.230805 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.230718 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:28.271201 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.271169 2560 generic.go:356] "Generic (PLEG): container finished" podID="f5c09135-a711-4e32-8327-84101b838752" containerID="d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d" exitCode=0 Jul 07 14:12:28.271201 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.271190 2560 generic.go:356] "Generic (PLEG): container finished" podID="f5c09135-a711-4e32-8327-84101b838752" containerID="c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8" exitCode=0 Jul 07 14:12:28.271201 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.271199 2560 generic.go:356] "Generic (PLEG): container finished" podID="f5c09135-a711-4e32-8327-84101b838752" containerID="6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac" exitCode=0 Jul 07 14:12:28.271201 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.271206 2560 generic.go:356] "Generic (PLEG): container finished" podID="f5c09135-a711-4e32-8327-84101b838752" containerID="6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a" exitCode=0 Jul 07 14:12:28.271453 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.271236 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c09135-a711-4e32-8327-84101b838752","Type":"ContainerDied","Data":"d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d"} Jul 07 14:12:28.271453 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.271315 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c09135-a711-4e32-8327-84101b838752","Type":"ContainerDied","Data":"c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8"} Jul 07 14:12:28.271453 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.271360 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c09135-a711-4e32-8327-84101b838752","Type":"ContainerDied","Data":"6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac"} Jul 07 14:12:28.271453 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.271399 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c09135-a711-4e32-8327-84101b838752","Type":"ContainerDied","Data":"6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a"} Jul 07 14:12:28.272662 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.272642 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67554b6fdf-zd82b" event={"ID":"c35722dd-f496-44ca-b846-46687b0f15dd","Type":"ContainerStarted","Data":"273981c12ee6c4f9a036d67dc0a57db6dbfe6ff0b8577cd5486545e385de0b5c"} Jul 07 14:12:28.272773 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.272678 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67554b6fdf-zd82b" event={"ID":"c35722dd-f496-44ca-b846-46687b0f15dd","Type":"ContainerStarted","Data":"ae9974052c1108bf559f75c1534bea8440a0da2d829f81356f587bba832fbfbb"} Jul 07 14:12:28.290996 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.290959 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67554b6fdf-zd82b" podStartSLOduration=1.290943795 podStartE2EDuration="1.290943795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:12:28.289233151 +0000 UTC m=+306.721475655" watchObservedRunningTime="2026-07-07 14:12:28.290943795 +0000 UTC m=+306.723186299" Jul 07 14:12:28.645129 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.645098 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:28.678335 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.678309 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f5c09135-a711-4e32-8327-84101b838752-alertmanager-main-db\") pod \"f5c09135-a711-4e32-8327-84101b838752\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " Jul 07 14:12:28.678506 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.678344 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy\") pod \"f5c09135-a711-4e32-8327-84101b838752\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " Jul 07 14:12:28.678506 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.678366 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-web-config\") pod \"f5c09135-a711-4e32-8327-84101b838752\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " Jul 07 14:12:28.678506 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.678384 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-cluster-tls-config\") pod \"f5c09135-a711-4e32-8327-84101b838752\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " Jul 07 14:12:28.678506 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.678407 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f5c09135-a711-4e32-8327-84101b838752\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " Jul 07 14:12:28.678506 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.678424 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy-web\") pod \"f5c09135-a711-4e32-8327-84101b838752\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " Jul 07 14:12:28.678506 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.678439 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5c09135-a711-4e32-8327-84101b838752-config-out\") pod \"f5c09135-a711-4e32-8327-84101b838752\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " Jul 07 14:12:28.678506 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.678461 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c09135-a711-4e32-8327-84101b838752-metrics-client-ca\") pod \"f5c09135-a711-4e32-8327-84101b838752\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " Jul 07 14:12:28.678506 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.678486 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-main-tls\") pod \"f5c09135-a711-4e32-8327-84101b838752\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " Jul 07 14:12:28.678921 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.678514 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-config-volume\") pod \"f5c09135-a711-4e32-8327-84101b838752\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " Jul 07 14:12:28.678921 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.678537 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlhfp\" (UniqueName: \"kubernetes.io/projected/f5c09135-a711-4e32-8327-84101b838752-kube-api-access-nlhfp\") pod \"f5c09135-a711-4e32-8327-84101b838752\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " Jul 07 14:12:28.678921 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.678567 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c09135-a711-4e32-8327-84101b838752-alertmanager-trusted-ca-bundle\") pod \"f5c09135-a711-4e32-8327-84101b838752\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " Jul 07 14:12:28.678921 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.678607 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5c09135-a711-4e32-8327-84101b838752-tls-assets\") pod \"f5c09135-a711-4e32-8327-84101b838752\" (UID: \"f5c09135-a711-4e32-8327-84101b838752\") " Jul 07 14:12:28.678921 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.678665 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c09135-a711-4e32-8327-84101b838752-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f5c09135-a711-4e32-8327-84101b838752" (UID: "f5c09135-a711-4e32-8327-84101b838752"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:12:28.680715 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.680682 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c09135-a711-4e32-8327-84101b838752-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f5c09135-a711-4e32-8327-84101b838752" (UID: "f5c09135-a711-4e32-8327-84101b838752"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:12:28.682052 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.682006 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5c09135-a711-4e32-8327-84101b838752-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:28.682160 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.682034 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f5c09135-a711-4e32-8327-84101b838752-alertmanager-main-db\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:28.682774 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.682343 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c09135-a711-4e32-8327-84101b838752-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f5c09135-a711-4e32-8327-84101b838752" (UID: "f5c09135-a711-4e32-8327-84101b838752"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:12:28.683000 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.682980 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c09135-a711-4e32-8327-84101b838752-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f5c09135-a711-4e32-8327-84101b838752" (UID: "f5c09135-a711-4e32-8327-84101b838752"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:12:28.683152 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.683133 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c09135-a711-4e32-8327-84101b838752-kube-api-access-nlhfp" (OuterVolumeSpecName: "kube-api-access-nlhfp") pod "f5c09135-a711-4e32-8327-84101b838752" (UID: "f5c09135-a711-4e32-8327-84101b838752"). InnerVolumeSpecName "kube-api-access-nlhfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:12:28.683382 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.683354 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f5c09135-a711-4e32-8327-84101b838752" (UID: "f5c09135-a711-4e32-8327-84101b838752"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:12:28.684394 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.684342 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-config-volume" (OuterVolumeSpecName: "config-volume") pod "f5c09135-a711-4e32-8327-84101b838752" (UID: "f5c09135-a711-4e32-8327-84101b838752"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:12:28.684394 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.684343 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f5c09135-a711-4e32-8327-84101b838752" (UID: "f5c09135-a711-4e32-8327-84101b838752"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:12:28.684557 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.684445 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f5c09135-a711-4e32-8327-84101b838752" (UID: "f5c09135-a711-4e32-8327-84101b838752"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:12:28.685220 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.685167 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c09135-a711-4e32-8327-84101b838752-config-out" (OuterVolumeSpecName: "config-out") pod "f5c09135-a711-4e32-8327-84101b838752" (UID: "f5c09135-a711-4e32-8327-84101b838752"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:12:28.685830 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.685808 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f5c09135-a711-4e32-8327-84101b838752" (UID: "f5c09135-a711-4e32-8327-84101b838752"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:12:28.688239 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.688206 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f5c09135-a711-4e32-8327-84101b838752" (UID: "f5c09135-a711-4e32-8327-84101b838752"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:12:28.693957 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.693926 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-web-config" (OuterVolumeSpecName: "web-config") pod "f5c09135-a711-4e32-8327-84101b838752" (UID: "f5c09135-a711-4e32-8327-84101b838752"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:12:28.783168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.783103 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:28.783168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.783124 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:28.783168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.783135 2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f5c09135-a711-4e32-8327-84101b838752-config-out\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:28.783168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.783144 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5c09135-a711-4e32-8327-84101b838752-metrics-client-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:28.783168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.783159 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-main-tls\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:28.783168 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.783168 2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-config-volume\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:28.783446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.783176 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlhfp\" (UniqueName: \"kubernetes.io/projected/f5c09135-a711-4e32-8327-84101b838752-kube-api-access-nlhfp\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:28.783446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.783184 2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f5c09135-a711-4e32-8327-84101b838752-tls-assets\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:28.783446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.783192 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:28.783446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.783201 2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-web-config\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:28.783446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:28.783209 2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f5c09135-a711-4e32-8327-84101b838752-cluster-tls-config\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:12:29.279827 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.279791 2560 generic.go:356] "Generic (PLEG): container finished" podID="f5c09135-a711-4e32-8327-84101b838752" containerID="a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79" exitCode=0 Jul 07 14:12:29.279827 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.279820 2560 generic.go:356] "Generic (PLEG): container finished" podID="f5c09135-a711-4e32-8327-84101b838752" containerID="a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32" exitCode=0 Jul 07 14:12:29.280048 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.279846 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c09135-a711-4e32-8327-84101b838752","Type":"ContainerDied","Data":"a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79"} Jul 07 14:12:29.280048 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.279893 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.280048 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.279927 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c09135-a711-4e32-8327-84101b838752","Type":"ContainerDied","Data":"a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32"} Jul 07 14:12:29.280048 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.279959 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f5c09135-a711-4e32-8327-84101b838752","Type":"ContainerDied","Data":"638627b838bcbc8d679851ba5b9388486bcf982b734cbf604396f1c6c3948603"} Jul 07 14:12:29.280048 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.279999 2560 scope.go:117] "RemoveContainer" containerID="d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d" Jul 07 14:12:29.288815 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.288800 2560 scope.go:117] "RemoveContainer" containerID="a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79" Jul 07 14:12:29.295764 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.295745 2560 scope.go:117] "RemoveContainer" containerID="c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8" Jul 07 14:12:29.302930 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.302910 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:12:29.304355 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.304337 2560 scope.go:117] "RemoveContainer" containerID="a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32" Jul 07 14:12:29.307965 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.307944 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:12:29.311193 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.311178 2560 scope.go:117] "RemoveContainer" containerID="6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac" Jul 07 14:12:29.318044 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.318025 2560 scope.go:117] "RemoveContainer" containerID="6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a" Jul 07 14:12:29.325054 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.325030 2560 scope.go:117] "RemoveContainer" containerID="adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c" Jul 07 14:12:29.331872 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.331856 2560 scope.go:117] "RemoveContainer" containerID="d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d" Jul 07 14:12:29.332108 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:12:29.332090 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d\": container with ID starting with d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d not found: ID does not exist" containerID="d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d" Jul 07 14:12:29.332149 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.332116 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d"} err="failed to get container status \"d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d\": rpc error: code = NotFound desc = could not find container \"d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d\": container with ID starting with d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d not found: ID does not exist" Jul 07 14:12:29.332149 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.332134 2560 scope.go:117] "RemoveContainer" containerID="a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79" Jul 07 14:12:29.332425 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:12:29.332406 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79\": container with ID starting with a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79 not found: ID does not exist" containerID="a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79" Jul 07 14:12:29.332480 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.332432 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79"} err="failed to get container status \"a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79\": rpc error: code = NotFound desc = could not find container \"a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79\": container with ID starting with a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79 not found: ID does not exist" Jul 07 14:12:29.332480 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.332452 2560 scope.go:117] "RemoveContainer" containerID="c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8" Jul 07 14:12:29.332689 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:12:29.332675 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8\": container with ID starting with c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8 not found: ID does not exist" containerID="c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8" Jul 07 14:12:29.332724 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.332692 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8"} err="failed to get container status \"c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8\": rpc error: code = NotFound desc = could not find container \"c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8\": container with ID starting with c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8 not found: ID does not exist" Jul 07 14:12:29.332724 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.332705 2560 scope.go:117] "RemoveContainer" containerID="a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32" Jul 07 14:12:29.332919 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:12:29.332903 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32\": container with ID starting with a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32 not found: ID does not exist" containerID="a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32" Jul 07 14:12:29.332953 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.332924 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32"} err="failed to get container status \"a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32\": rpc error: code = NotFound desc = could not find container \"a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32\": container with ID starting with a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32 not found: ID does not exist" Jul 07 14:12:29.332953 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.332939 2560 scope.go:117] "RemoveContainer" containerID="6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac" Jul 07 14:12:29.333159 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:12:29.333145 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac\": container with ID starting with 6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac not found: ID does not exist" containerID="6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac" Jul 07 14:12:29.333198 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.333162 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac"} err="failed to get container status \"6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac\": rpc error: code = NotFound desc = could not find container \"6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac\": container with ID starting with 6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac not found: ID does not exist" Jul 07 14:12:29.333198 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.333175 2560 scope.go:117] "RemoveContainer" containerID="6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a" Jul 07 14:12:29.333643 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:12:29.333515 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a\": container with ID starting with 6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a not found: ID does not exist" containerID="6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a" Jul 07 14:12:29.333643 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.333568 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a"} err="failed to get container status \"6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a\": rpc error: code = NotFound desc = could not find container \"6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a\": container with ID starting with 6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a not found: ID does not exist" Jul 07 14:12:29.333643 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.333587 2560 scope.go:117] "RemoveContainer" containerID="adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c" Jul 07 14:12:29.334181 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:12:29.334158 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c\": container with ID starting with adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c not found: ID does not exist" containerID="adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c" Jul 07 14:12:29.334255 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.334187 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c"} err="failed to get container status \"adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c\": rpc error: code = NotFound desc = could not find container \"adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c\": container with ID starting with adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c not found: ID does not exist" Jul 07 14:12:29.334255 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.334207 2560 scope.go:117] "RemoveContainer" containerID="d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d" Jul 07 14:12:29.334679 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.334657 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d"} err="failed to get container status \"d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d\": rpc error: code = NotFound desc = could not find container \"d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d\": container with ID starting with d70c0982bf812a025e2c4a5e402ce52bc77022c05cc5a96ca8b09889c5822f3d not found: ID does not exist" Jul 07 14:12:29.334753 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.334681 2560 scope.go:117] "RemoveContainer" containerID="a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79" Jul 07 14:12:29.334941 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.334921 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79"} err="failed to get container status \"a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79\": rpc error: code = NotFound desc = could not find container \"a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79\": container with ID starting with a0ed440b15b7a875eed65720b8267ae0095792ddef5a1a168fa6ce1cf6e17a79 not found: ID does not exist" Jul 07 14:12:29.335020 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.334943 2560 scope.go:117] "RemoveContainer" containerID="c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8" Jul 07 14:12:29.335203 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335185 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8"} err="failed to get container status \"c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8\": rpc error: code = NotFound desc = could not find container \"c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8\": container with ID starting with c605949ef5d01dc92a65ad3021f33fa4945548a830879c082e1471d892b169b8 not found: ID does not exist" Jul 07 14:12:29.335301 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335204 2560 scope.go:117] "RemoveContainer" containerID="a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32" Jul 07 14:12:29.335351 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335341 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:12:29.335447 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335429 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32"} err="failed to get container status \"a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32\": rpc error: code = NotFound desc = could not find container \"a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32\": container with ID starting with a25bc4da4977b275a2cd75f74e9d87c6051f447dd919604c4629aca1f7a0bd32 not found: ID does not exist" Jul 07 14:12:29.335513 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335448 2560 scope.go:117] "RemoveContainer" containerID="6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac" Jul 07 14:12:29.335640 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335628 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="kube-rbac-proxy-metric" Jul 07 14:12:29.335702 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335643 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="kube-rbac-proxy-metric" Jul 07 14:12:29.335702 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335642 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac"} err="failed to get container status \"6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac\": rpc error: code = NotFound desc = could not find container \"6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac\": container with ID starting with 6767caf2a1b2ca743f81f118c592d9381dfef4929682e7abbce144b507d4b3ac not found: ID does not exist" Jul 07 14:12:29.335702 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335654 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="kube-rbac-proxy" Jul 07 14:12:29.335702 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335657 2560 scope.go:117] "RemoveContainer" containerID="6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a" Jul 07 14:12:29.335702 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335660 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="kube-rbac-proxy" Jul 07 14:12:29.335702 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335668 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="prom-label-proxy" Jul 07 14:12:29.335702 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335673 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="prom-label-proxy" Jul 07 14:12:29.335702 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335681 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="alertmanager" Jul 07 14:12:29.335702 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335690 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="alertmanager" Jul 07 14:12:29.335702 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335695 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="config-reloader" Jul 07 14:12:29.335702 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335700 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="config-reloader" Jul 07 14:12:29.336073 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335712 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="kube-rbac-proxy-web" Jul 07 14:12:29.336073 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335720 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="kube-rbac-proxy-web" Jul 07 14:12:29.336073 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335734 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="init-config-reloader" Jul 07 14:12:29.336073 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335742 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="init-config-reloader" Jul 07 14:12:29.336073 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335804 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="kube-rbac-proxy-web" Jul 07 14:12:29.336073 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335813 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="alertmanager" Jul 07 14:12:29.336073 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335818 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="kube-rbac-proxy" Jul 07 14:12:29.336073 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335823 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="kube-rbac-proxy-metric" Jul 07 14:12:29.336073 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335829 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="prom-label-proxy" Jul 07 14:12:29.336073 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335836 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c09135-a711-4e32-8327-84101b838752" containerName="config-reloader" Jul 07 14:12:29.336073 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335848 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a"} err="failed to get container status \"6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a\": rpc error: code = NotFound desc = could not find container \"6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a\": container with ID starting with 6fc10c61193ece44a7e513779ef79ccae5b09111312527727e47af3a5a13149a not found: ID does not exist" Jul 07 14:12:29.336073 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.335862 2560 scope.go:117] "RemoveContainer" containerID="adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c" Jul 07 14:12:29.336479 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.336074 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c"} err="failed to get container status \"adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c\": rpc error: code = NotFound desc = could not find container \"adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c\": container with ID starting with adb5af4dee5056056d16693bc1a90492197701d58de86b7f6e325fe2f82a171c not found: ID does not exist" Jul 07 14:12:29.339495 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.339480 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.341673 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.341657 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 14:12:29.341762 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.341661 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 14:12:29.342011 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.341994 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 14:12:29.342082 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.342057 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 14:12:29.342082 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.342065 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 14:12:29.342231 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.342213 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 14:12:29.342320 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.342238 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 14:12:29.342320 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.342241 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 14:12:29.342727 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.342713 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f6wdq\"" Jul 07 14:12:29.347325 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.347310 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 14:12:29.353488 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.353468 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:12:29.388296 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.388251 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6bced0b2-a257-435c-ae29-b93f0d8e35e4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.388414 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.388305 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.388414 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.388326 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4thr\" (UniqueName: \"kubernetes.io/projected/6bced0b2-a257-435c-ae29-b93f0d8e35e4-kube-api-access-l4thr\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.388414 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.388342 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6bced0b2-a257-435c-ae29-b93f0d8e35e4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.388414 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.388359 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-web-config\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.388414 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.388401 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6bced0b2-a257-435c-ae29-b93f0d8e35e4-config-out\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.388568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.388433 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.388568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.388479 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.388568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.388516 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.388568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.388554 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-config-volume\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.388680 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.388594 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.388680 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.388612 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bced0b2-a257-435c-ae29-b93f0d8e35e4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.388680 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.388637 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bced0b2-a257-435c-ae29-b93f0d8e35e4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.489372 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.489331 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6bced0b2-a257-435c-ae29-b93f0d8e35e4-config-out\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.489372 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.489377 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.489559 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.489403 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.489559 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.489456 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.489559 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.489485 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-config-volume\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.489559 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.489520 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.489559 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.489539 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bced0b2-a257-435c-ae29-b93f0d8e35e4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.489762 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.489574 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bced0b2-a257-435c-ae29-b93f0d8e35e4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.489762 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.489596 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6bced0b2-a257-435c-ae29-b93f0d8e35e4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.489762 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.489631 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.489762 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.489659 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4thr\" (UniqueName: \"kubernetes.io/projected/6bced0b2-a257-435c-ae29-b93f0d8e35e4-kube-api-access-l4thr\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.489762 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.489688 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6bced0b2-a257-435c-ae29-b93f0d8e35e4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.489762 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.489716 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-web-config\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.492657 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.492580 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6bced0b2-a257-435c-ae29-b93f0d8e35e4-config-out\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.492824 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.492803 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.492824 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.492819 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-config-volume\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.492937 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.492835 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6bced0b2-a257-435c-ae29-b93f0d8e35e4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.492937 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.492872 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.493043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.492977 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.493296 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.493246 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6bced0b2-a257-435c-ae29-b93f0d8e35e4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.493359 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.493256 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-web-config\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.493521 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.493491 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.493593 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.493569 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6bced0b2-a257-435c-ae29-b93f0d8e35e4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.493650 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.493578 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bced0b2-a257-435c-ae29-b93f0d8e35e4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.494863 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.494839 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6bced0b2-a257-435c-ae29-b93f0d8e35e4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.501007 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.500987 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4thr\" (UniqueName: \"kubernetes.io/projected/6bced0b2-a257-435c-ae29-b93f0d8e35e4-kube-api-access-l4thr\") pod \"alertmanager-main-0\" (UID: \"6bced0b2-a257-435c-ae29-b93f0d8e35e4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.649290 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.649234 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:12:29.790216 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:29.790178 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:12:29.793837 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:12:29.793808 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bced0b2_a257_435c_ae29_b93f0d8e35e4.slice/crio-08ad3eec37756a2550405f9c9c012db2ab15f675fcaff87587416df23dbdfb32 WatchSource:0}: Error finding container 08ad3eec37756a2550405f9c9c012db2ab15f675fcaff87587416df23dbdfb32: Status 404 returned error can't find the container with id 08ad3eec37756a2550405f9c9c012db2ab15f675fcaff87587416df23dbdfb32 Jul 07 14:12:30.208401 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:30.208370 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5c09135-a711-4e32-8327-84101b838752" path="/var/lib/kubelet/pods/f5c09135-a711-4e32-8327-84101b838752/volumes" Jul 07 14:12:30.285672 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:30.285640 2560 generic.go:356] "Generic (PLEG): container finished" podID="6bced0b2-a257-435c-ae29-b93f0d8e35e4" containerID="1b3f216175d6237f8fe7ea7cf33088cb08b498de3e99feb110ef7d82c83d8c42" exitCode=0 Jul 07 14:12:30.285812 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:30.285690 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bced0b2-a257-435c-ae29-b93f0d8e35e4","Type":"ContainerDied","Data":"1b3f216175d6237f8fe7ea7cf33088cb08b498de3e99feb110ef7d82c83d8c42"} Jul 07 14:12:30.285812 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:30.285751 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bced0b2-a257-435c-ae29-b93f0d8e35e4","Type":"ContainerStarted","Data":"08ad3eec37756a2550405f9c9c012db2ab15f675fcaff87587416df23dbdfb32"} Jul 07 14:12:30.873871 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:30.873840 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:30.973366 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:30.973310 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:31.297372 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:31.297338 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bced0b2-a257-435c-ae29-b93f0d8e35e4","Type":"ContainerStarted","Data":"da30dd9061fdd4ba6ccf4a7d50bedfba70824dad004307e7edb5e48f40f4602f"} Jul 07 14:12:31.297535 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:31.297440 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bced0b2-a257-435c-ae29-b93f0d8e35e4","Type":"ContainerStarted","Data":"c543b6abac976723f62e838e0fc7585d15f48b7390a8155bdb09afead138061f"} Jul 07 14:12:31.297535 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:31.297481 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bced0b2-a257-435c-ae29-b93f0d8e35e4","Type":"ContainerStarted","Data":"8fb318c06f8314201c2ea59654c57c2a868b8c70d82ee8c0d3ff65ab50c9dc58"} Jul 07 14:12:31.297535 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:31.297517 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bced0b2-a257-435c-ae29-b93f0d8e35e4","Type":"ContainerStarted","Data":"4963e9f588e4981b6c7cebf0b92897203d20e49f8a3c3c6344f55e65764e19d9"} Jul 07 14:12:31.297713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:31.297558 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bced0b2-a257-435c-ae29-b93f0d8e35e4","Type":"ContainerStarted","Data":"3d2ca3c9421f2c45e049eb1093c1551905dc7edbb603d8f21876a2b707c1a97e"} Jul 07 14:12:31.297713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:31.297600 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6bced0b2-a257-435c-ae29-b93f0d8e35e4","Type":"ContainerStarted","Data":"b76bec572f06de05640797d764927622136b2862aee46038aa76812d607cc46a"} Jul 07 14:12:31.333239 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:31.333194 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.317873028 podStartE2EDuration="2.333179623s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.306595ms" isStatefulPod=false podCreationTimestamp="2026-07-07 14:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:12:31.331423933 +0000 UTC m=+309.763666435" watchObservedRunningTime="2026-07-07 14:12:31.333179623 +0000 UTC m=+309.765422124" Jul 07 14:12:33.388222 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:33.388192 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:33.457563 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:33.457529 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:37.763137 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:37.763099 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:37.763696 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:37.763254 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:37.767917 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:37.767895 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:38.329628 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:38.329604 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:12:38.379987 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:38.379953 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65d5f7444-pl479"] Jul 07 14:12:38.514014 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:38.513982 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:38.614871 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:38.614795 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:38.665304 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:38.665275 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:49.982106 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:49.982074 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:12:50.080371 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:12:50.080338 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:03.400901 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.400862 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-65d5f7444-pl479" podUID="944599fc-10ae-4a3a-850a-8206bb28f61b" containerName="console" containerID="cri-o://53453a2cbea1c7ef54051c4d8e407703363cdde7cecc61466e17038781aa2792" gracePeriod=15 Jul 07 14:13:03.636857 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.636833 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:13:03.771580 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.771484 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/944599fc-10ae-4a3a-850a-8206bb28f61b-console-oauth-config\") pod \"944599fc-10ae-4a3a-850a-8206bb28f61b\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " Jul 07 14:13:03.771580 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.771524 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hks5t\" (UniqueName: \"kubernetes.io/projected/944599fc-10ae-4a3a-850a-8206bb28f61b-kube-api-access-hks5t\") pod \"944599fc-10ae-4a3a-850a-8206bb28f61b\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " Jul 07 14:13:03.771580 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.771543 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-oauth-serving-cert\") pod \"944599fc-10ae-4a3a-850a-8206bb28f61b\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " Jul 07 14:13:03.771580 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.771562 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/944599fc-10ae-4a3a-850a-8206bb28f61b-console-serving-cert\") pod \"944599fc-10ae-4a3a-850a-8206bb28f61b\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " Jul 07 14:13:03.771915 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.771593 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-service-ca\") pod \"944599fc-10ae-4a3a-850a-8206bb28f61b\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " Jul 07 14:13:03.771915 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.771651 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-trusted-ca-bundle\") pod \"944599fc-10ae-4a3a-850a-8206bb28f61b\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " Jul 07 14:13:03.771915 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.771692 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-console-config\") pod \"944599fc-10ae-4a3a-850a-8206bb28f61b\" (UID: \"944599fc-10ae-4a3a-850a-8206bb28f61b\") " Jul 07 14:13:03.772076 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.771915 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "944599fc-10ae-4a3a-850a-8206bb28f61b" (UID: "944599fc-10ae-4a3a-850a-8206bb28f61b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:13:03.772203 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.772180 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-console-config" (OuterVolumeSpecName: "console-config") pod "944599fc-10ae-4a3a-850a-8206bb28f61b" (UID: "944599fc-10ae-4a3a-850a-8206bb28f61b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:13:03.772203 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.772191 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-service-ca" (OuterVolumeSpecName: "service-ca") pod "944599fc-10ae-4a3a-850a-8206bb28f61b" (UID: "944599fc-10ae-4a3a-850a-8206bb28f61b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:13:03.772596 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.772220 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "944599fc-10ae-4a3a-850a-8206bb28f61b" (UID: "944599fc-10ae-4a3a-850a-8206bb28f61b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:13:03.773936 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.773911 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944599fc-10ae-4a3a-850a-8206bb28f61b-kube-api-access-hks5t" (OuterVolumeSpecName: "kube-api-access-hks5t") pod "944599fc-10ae-4a3a-850a-8206bb28f61b" (UID: "944599fc-10ae-4a3a-850a-8206bb28f61b"). InnerVolumeSpecName "kube-api-access-hks5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:13:03.773936 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.773920 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/944599fc-10ae-4a3a-850a-8206bb28f61b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "944599fc-10ae-4a3a-850a-8206bb28f61b" (UID: "944599fc-10ae-4a3a-850a-8206bb28f61b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:13:03.774077 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.773963 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/944599fc-10ae-4a3a-850a-8206bb28f61b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "944599fc-10ae-4a3a-850a-8206bb28f61b" (UID: "944599fc-10ae-4a3a-850a-8206bb28f61b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:13:03.873082 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.873046 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/944599fc-10ae-4a3a-850a-8206bb28f61b-console-oauth-config\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:13:03.873082 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.873076 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hks5t\" (UniqueName: \"kubernetes.io/projected/944599fc-10ae-4a3a-850a-8206bb28f61b-kube-api-access-hks5t\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:13:03.873082 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.873089 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-oauth-serving-cert\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:13:03.873330 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.873103 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/944599fc-10ae-4a3a-850a-8206bb28f61b-console-serving-cert\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:13:03.873330 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.873115 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-service-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:13:03.873330 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.873126 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-trusted-ca-bundle\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:13:03.873330 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:03.873136 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/944599fc-10ae-4a3a-850a-8206bb28f61b-console-config\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:13:04.430886 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:04.430852 2560 generic.go:356] "Generic (PLEG): container finished" podID="944599fc-10ae-4a3a-850a-8206bb28f61b" containerID="53453a2cbea1c7ef54051c4d8e407703363cdde7cecc61466e17038781aa2792" exitCode=0 Jul 07 14:13:04.431257 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:04.430893 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d5f7444-pl479" Jul 07 14:13:04.431257 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:04.430910 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d5f7444-pl479" event={"ID":"944599fc-10ae-4a3a-850a-8206bb28f61b","Type":"ContainerDied","Data":"53453a2cbea1c7ef54051c4d8e407703363cdde7cecc61466e17038781aa2792"} Jul 07 14:13:04.431257 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:04.430954 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d5f7444-pl479" event={"ID":"944599fc-10ae-4a3a-850a-8206bb28f61b","Type":"ContainerDied","Data":"d94cc1cd98c77055d3ed201d915f1703e4a1b7a1dfbd8215f07fca3dd253dee3"} Jul 07 14:13:04.431257 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:04.430973 2560 scope.go:117] "RemoveContainer" containerID="53453a2cbea1c7ef54051c4d8e407703363cdde7cecc61466e17038781aa2792" Jul 07 14:13:04.439333 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:04.439314 2560 scope.go:117] "RemoveContainer" containerID="53453a2cbea1c7ef54051c4d8e407703363cdde7cecc61466e17038781aa2792" Jul 07 14:13:04.439597 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:13:04.439575 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53453a2cbea1c7ef54051c4d8e407703363cdde7cecc61466e17038781aa2792\": container with ID starting with 53453a2cbea1c7ef54051c4d8e407703363cdde7cecc61466e17038781aa2792 not found: ID does not exist" containerID="53453a2cbea1c7ef54051c4d8e407703363cdde7cecc61466e17038781aa2792" Jul 07 14:13:04.439642 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:04.439606 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53453a2cbea1c7ef54051c4d8e407703363cdde7cecc61466e17038781aa2792"} err="failed to get container status \"53453a2cbea1c7ef54051c4d8e407703363cdde7cecc61466e17038781aa2792\": rpc error: code = NotFound desc = could not find container \"53453a2cbea1c7ef54051c4d8e407703363cdde7cecc61466e17038781aa2792\": container with ID starting with 53453a2cbea1c7ef54051c4d8e407703363cdde7cecc61466e17038781aa2792 not found: ID does not exist" Jul 07 14:13:04.447577 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:04.447550 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65d5f7444-pl479"] Jul 07 14:13:04.448762 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:04.448740 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65d5f7444-pl479"] Jul 07 14:13:06.207379 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:06.207348 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="944599fc-10ae-4a3a-850a-8206bb28f61b" path="/var/lib/kubelet/pods/944599fc-10ae-4a3a-850a-8206bb28f61b/volumes" Jul 07 14:13:13.790905 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:13.790809 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:13.901479 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:13.901448 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:14.563188 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.563157 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl"] Jul 07 14:13:14.563473 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.563461 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="944599fc-10ae-4a3a-850a-8206bb28f61b" containerName="console" Jul 07 14:13:14.563534 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.563475 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="944599fc-10ae-4a3a-850a-8206bb28f61b" containerName="console" Jul 07 14:13:14.563572 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.563536 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="944599fc-10ae-4a3a-850a-8206bb28f61b" containerName="console" Jul 07 14:13:14.566037 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.566020 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" Jul 07 14:13:14.569204 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.569175 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 14:13:14.569368 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.569181 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 14:13:14.569915 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.569899 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5nmlx\"" Jul 07 14:13:14.578404 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.578378 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl"] Jul 07 14:13:14.663901 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.663861 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/414f071f-cb41-4ad2-9eaf-3d2504a207bb-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl\" (UID: \"414f071f-cb41-4ad2-9eaf-3d2504a207bb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" Jul 07 14:13:14.663901 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.663901 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/414f071f-cb41-4ad2-9eaf-3d2504a207bb-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl\" (UID: \"414f071f-cb41-4ad2-9eaf-3d2504a207bb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" Jul 07 14:13:14.664118 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.663935 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fw7b\" (UniqueName: \"kubernetes.io/projected/414f071f-cb41-4ad2-9eaf-3d2504a207bb-kube-api-access-5fw7b\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl\" (UID: \"414f071f-cb41-4ad2-9eaf-3d2504a207bb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" Jul 07 14:13:14.765176 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.765135 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fw7b\" (UniqueName: \"kubernetes.io/projected/414f071f-cb41-4ad2-9eaf-3d2504a207bb-kube-api-access-5fw7b\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl\" (UID: \"414f071f-cb41-4ad2-9eaf-3d2504a207bb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" Jul 07 14:13:14.765362 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.765223 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/414f071f-cb41-4ad2-9eaf-3d2504a207bb-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl\" (UID: \"414f071f-cb41-4ad2-9eaf-3d2504a207bb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" Jul 07 14:13:14.765362 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.765246 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/414f071f-cb41-4ad2-9eaf-3d2504a207bb-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl\" (UID: \"414f071f-cb41-4ad2-9eaf-3d2504a207bb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" Jul 07 14:13:14.765628 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.765614 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/414f071f-cb41-4ad2-9eaf-3d2504a207bb-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl\" (UID: \"414f071f-cb41-4ad2-9eaf-3d2504a207bb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" Jul 07 14:13:14.765665 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.765646 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/414f071f-cb41-4ad2-9eaf-3d2504a207bb-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl\" (UID: \"414f071f-cb41-4ad2-9eaf-3d2504a207bb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" Jul 07 14:13:14.774140 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.774115 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fw7b\" (UniqueName: \"kubernetes.io/projected/414f071f-cb41-4ad2-9eaf-3d2504a207bb-kube-api-access-5fw7b\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl\" (UID: \"414f071f-cb41-4ad2-9eaf-3d2504a207bb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" Jul 07 14:13:14.875208 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:14.875118 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" Jul 07 14:13:15.012736 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:15.012709 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl"] Jul 07 14:13:15.015148 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:13:15.015116 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414f071f_cb41_4ad2_9eaf_3d2504a207bb.slice/crio-6e87c6c667e569d2776d57d2fb66b722f74e758d363a1e4308f9a43b7533acaa WatchSource:0}: Error finding container 6e87c6c667e569d2776d57d2fb66b722f74e758d363a1e4308f9a43b7533acaa: Status 404 returned error can't find the container with id 6e87c6c667e569d2776d57d2fb66b722f74e758d363a1e4308f9a43b7533acaa Jul 07 14:13:15.017110 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:15.017095 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:13:15.473415 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:15.473386 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" event={"ID":"414f071f-cb41-4ad2-9eaf-3d2504a207bb","Type":"ContainerStarted","Data":"6e87c6c667e569d2776d57d2fb66b722f74e758d363a1e4308f9a43b7533acaa"} Jul 07 14:13:20.183899 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:20.183861 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:20.239252 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:20.239219 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:23.508755 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:23.508718 2560 generic.go:356] "Generic (PLEG): container finished" podID="414f071f-cb41-4ad2-9eaf-3d2504a207bb" containerID="ef77242e07adfbcaa13c7dad4c7e60742d9e6db4064aff23e1853587c66ded34" exitCode=0 Jul 07 14:13:23.509163 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:23.508808 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" event={"ID":"414f071f-cb41-4ad2-9eaf-3d2504a207bb","Type":"ContainerDied","Data":"ef77242e07adfbcaa13c7dad4c7e60742d9e6db4064aff23e1853587c66ded34"} Jul 07 14:13:25.518811 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:25.518771 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" event={"ID":"414f071f-cb41-4ad2-9eaf-3d2504a207bb","Type":"ContainerStarted","Data":"7529cfa22f986505f6c3f79fbc157200bc3c46d99964f861a0a22c118f4092d5"} Jul 07 14:13:26.524116 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:26.524078 2560 generic.go:356] "Generic (PLEG): container finished" podID="414f071f-cb41-4ad2-9eaf-3d2504a207bb" containerID="7529cfa22f986505f6c3f79fbc157200bc3c46d99964f861a0a22c118f4092d5" exitCode=0 Jul 07 14:13:26.524535 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:26.524161 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" event={"ID":"414f071f-cb41-4ad2-9eaf-3d2504a207bb","Type":"ContainerDied","Data":"7529cfa22f986505f6c3f79fbc157200bc3c46d99964f861a0a22c118f4092d5"} Jul 07 14:13:34.563325 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:34.563284 2560 generic.go:356] "Generic (PLEG): container finished" podID="414f071f-cb41-4ad2-9eaf-3d2504a207bb" containerID="cd2edc1b7555b11c5d121cb6a1d1c1c71f8c6a7934e6d55175d1b8b8e73d4b46" exitCode=0 Jul 07 14:13:34.563720 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:34.563359 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" event={"ID":"414f071f-cb41-4ad2-9eaf-3d2504a207bb","Type":"ContainerDied","Data":"cd2edc1b7555b11c5d121cb6a1d1c1c71f8c6a7934e6d55175d1b8b8e73d4b46"} Jul 07 14:13:35.703514 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:35.703491 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" Jul 07 14:13:35.755677 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:35.755649 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fw7b\" (UniqueName: \"kubernetes.io/projected/414f071f-cb41-4ad2-9eaf-3d2504a207bb-kube-api-access-5fw7b\") pod \"414f071f-cb41-4ad2-9eaf-3d2504a207bb\" (UID: \"414f071f-cb41-4ad2-9eaf-3d2504a207bb\") " Jul 07 14:13:35.755832 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:35.755762 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/414f071f-cb41-4ad2-9eaf-3d2504a207bb-util\") pod \"414f071f-cb41-4ad2-9eaf-3d2504a207bb\" (UID: \"414f071f-cb41-4ad2-9eaf-3d2504a207bb\") " Jul 07 14:13:35.755832 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:35.755792 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/414f071f-cb41-4ad2-9eaf-3d2504a207bb-bundle\") pod \"414f071f-cb41-4ad2-9eaf-3d2504a207bb\" (UID: \"414f071f-cb41-4ad2-9eaf-3d2504a207bb\") " Jul 07 14:13:35.756428 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:35.756404 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414f071f-cb41-4ad2-9eaf-3d2504a207bb-bundle" (OuterVolumeSpecName: "bundle") pod "414f071f-cb41-4ad2-9eaf-3d2504a207bb" (UID: "414f071f-cb41-4ad2-9eaf-3d2504a207bb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:13:35.757968 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:35.757946 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414f071f-cb41-4ad2-9eaf-3d2504a207bb-kube-api-access-5fw7b" (OuterVolumeSpecName: "kube-api-access-5fw7b") pod "414f071f-cb41-4ad2-9eaf-3d2504a207bb" (UID: "414f071f-cb41-4ad2-9eaf-3d2504a207bb"). InnerVolumeSpecName "kube-api-access-5fw7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:13:35.760844 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:35.760820 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/414f071f-cb41-4ad2-9eaf-3d2504a207bb-util" (OuterVolumeSpecName: "util") pod "414f071f-cb41-4ad2-9eaf-3d2504a207bb" (UID: "414f071f-cb41-4ad2-9eaf-3d2504a207bb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:13:35.856354 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:35.856241 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/414f071f-cb41-4ad2-9eaf-3d2504a207bb-util\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:13:35.856354 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:35.856300 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/414f071f-cb41-4ad2-9eaf-3d2504a207bb-bundle\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:13:35.856354 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:35.856309 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fw7b\" (UniqueName: \"kubernetes.io/projected/414f071f-cb41-4ad2-9eaf-3d2504a207bb-kube-api-access-5fw7b\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:13:36.573017 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:36.572984 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" event={"ID":"414f071f-cb41-4ad2-9eaf-3d2504a207bb","Type":"ContainerDied","Data":"6e87c6c667e569d2776d57d2fb66b722f74e758d363a1e4308f9a43b7533acaa"} Jul 07 14:13:36.573174 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:36.573027 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e87c6c667e569d2776d57d2fb66b722f74e758d363a1e4308f9a43b7533acaa" Jul 07 14:13:36.573174 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:36.573026 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675409nhxl" Jul 07 14:13:40.602250 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:40.602220 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:41.467635 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:41.467605 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:42.566697 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.566661 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t65q4"] Jul 07 14:13:42.567064 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.566953 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="414f071f-cb41-4ad2-9eaf-3d2504a207bb" containerName="util" Jul 07 14:13:42.567064 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.566963 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="414f071f-cb41-4ad2-9eaf-3d2504a207bb" containerName="util" Jul 07 14:13:42.567064 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.566971 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="414f071f-cb41-4ad2-9eaf-3d2504a207bb" containerName="pull" Jul 07 14:13:42.567064 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.566975 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="414f071f-cb41-4ad2-9eaf-3d2504a207bb" containerName="pull" Jul 07 14:13:42.567064 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.566987 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="414f071f-cb41-4ad2-9eaf-3d2504a207bb" containerName="extract" Jul 07 14:13:42.567064 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.566992 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="414f071f-cb41-4ad2-9eaf-3d2504a207bb" containerName="extract" Jul 07 14:13:42.567064 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.567040 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="414f071f-cb41-4ad2-9eaf-3d2504a207bb" containerName="extract" Jul 07 14:13:42.568998 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.568983 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t65q4" Jul 07 14:13:42.571345 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.571322 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vjpkc\"" Jul 07 14:13:42.571485 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.571407 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 07 14:13:42.571485 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.571477 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:13:42.581803 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.581779 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t65q4"] Jul 07 14:13:42.716174 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.716135 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gphc\" (UniqueName: \"kubernetes.io/projected/c1bd7569-7ee7-4244-bbbb-2cf5f78b9032-kube-api-access-7gphc\") pod \"cert-manager-operator-controller-manager-55c444bf86-t65q4\" (UID: \"c1bd7569-7ee7-4244-bbbb-2cf5f78b9032\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t65q4" Jul 07 14:13:42.716396 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.716195 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1bd7569-7ee7-4244-bbbb-2cf5f78b9032-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t65q4\" (UID: \"c1bd7569-7ee7-4244-bbbb-2cf5f78b9032\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t65q4" Jul 07 14:13:42.817594 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.817488 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gphc\" (UniqueName: \"kubernetes.io/projected/c1bd7569-7ee7-4244-bbbb-2cf5f78b9032-kube-api-access-7gphc\") pod \"cert-manager-operator-controller-manager-55c444bf86-t65q4\" (UID: \"c1bd7569-7ee7-4244-bbbb-2cf5f78b9032\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t65q4" Jul 07 14:13:42.817594 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.817550 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1bd7569-7ee7-4244-bbbb-2cf5f78b9032-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t65q4\" (UID: \"c1bd7569-7ee7-4244-bbbb-2cf5f78b9032\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t65q4" Jul 07 14:13:42.817890 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.817875 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1bd7569-7ee7-4244-bbbb-2cf5f78b9032-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t65q4\" (UID: \"c1bd7569-7ee7-4244-bbbb-2cf5f78b9032\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t65q4" Jul 07 14:13:42.827130 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.827101 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gphc\" (UniqueName: \"kubernetes.io/projected/c1bd7569-7ee7-4244-bbbb-2cf5f78b9032-kube-api-access-7gphc\") pod \"cert-manager-operator-controller-manager-55c444bf86-t65q4\" (UID: \"c1bd7569-7ee7-4244-bbbb-2cf5f78b9032\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t65q4" Jul 07 14:13:42.878644 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:42.878608 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t65q4" Jul 07 14:13:43.037155 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:43.037134 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t65q4"] Jul 07 14:13:43.041640 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:13:43.041613 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bd7569_7ee7_4244_bbbb_2cf5f78b9032.slice/crio-e60e1d12bd33729be5dfd19b0192ce165d46ba2a894fa837ff76775ca527d5e4 WatchSource:0}: Error finding container e60e1d12bd33729be5dfd19b0192ce165d46ba2a894fa837ff76775ca527d5e4: Status 404 returned error can't find the container with id e60e1d12bd33729be5dfd19b0192ce165d46ba2a894fa837ff76775ca527d5e4 Jul 07 14:13:43.602880 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:43.602847 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t65q4" event={"ID":"c1bd7569-7ee7-4244-bbbb-2cf5f78b9032","Type":"ContainerStarted","Data":"e60e1d12bd33729be5dfd19b0192ce165d46ba2a894fa837ff76775ca527d5e4"} Jul 07 14:13:44.339744 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:44.339708 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:44.441819 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:44.441776 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:45.614099 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:45.614063 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t65q4" event={"ID":"c1bd7569-7ee7-4244-bbbb-2cf5f78b9032","Type":"ContainerStarted","Data":"55b30b791ee85ca297f3d016b6e85092c651f09a18726757ae26b2b4fcf365cf"} Jul 07 14:13:45.640876 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:45.640824 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t65q4" podStartSLOduration=1.6105659129999998 podStartE2EDuration="3.640808054s" totalImagesPullingTime="2.030242141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:13:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:13:45.639528429 +0000 UTC m=+384.071770930" watchObservedRunningTime="2026-07-07 14:13:45.640808054 +0000 UTC m=+384.073050555" Jul 07 14:13:45.945060 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:45.944980 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:46.085772 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:46.085738 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:46.191574 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:46.191540 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:46.868207 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:46.868176 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2"] Jul 07 14:13:46.870794 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:46.870775 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" Jul 07 14:13:46.873770 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:46.873752 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 14:13:46.873998 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:46.873979 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5nmlx\"" Jul 07 14:13:46.874710 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:46.874692 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 14:13:46.881489 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:46.881464 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2"] Jul 07 14:13:46.953329 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:46.953290 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c24f\" (UniqueName: \"kubernetes.io/projected/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-kube-api-access-9c24f\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2\" (UID: \"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" Jul 07 14:13:46.953503 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:46.953362 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2\" (UID: \"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" Jul 07 14:13:46.953503 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:46.953383 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2\" (UID: \"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" Jul 07 14:13:47.054775 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:47.054726 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c24f\" (UniqueName: \"kubernetes.io/projected/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-kube-api-access-9c24f\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2\" (UID: \"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" Jul 07 14:13:47.054775 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:47.054789 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2\" (UID: \"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" Jul 07 14:13:47.054965 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:47.054807 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2\" (UID: \"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" Jul 07 14:13:47.055162 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:47.055146 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2\" (UID: \"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" Jul 07 14:13:47.055208 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:47.055191 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2\" (UID: \"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" Jul 07 14:13:47.063272 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:47.063244 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c24f\" (UniqueName: \"kubernetes.io/projected/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-kube-api-access-9c24f\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2\" (UID: \"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" Jul 07 14:13:47.179980 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:47.179881 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" Jul 07 14:13:47.313209 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:47.313182 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2"] Jul 07 14:13:47.315397 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:13:47.315369 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8e2b8a_ddd5_4dfc_ad86_c2664e372abe.slice/crio-8f43ebc4dc815bacaba512f7be1c65a29005a7d492d6d1a0c571c37f3e285884 WatchSource:0}: Error finding container 8f43ebc4dc815bacaba512f7be1c65a29005a7d492d6d1a0c571c37f3e285884: Status 404 returned error can't find the container with id 8f43ebc4dc815bacaba512f7be1c65a29005a7d492d6d1a0c571c37f3e285884 Jul 07 14:13:47.623360 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:47.623321 2560 generic.go:356] "Generic (PLEG): container finished" podID="dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe" containerID="478e717daaf58775d2cc3e2c35963d09b9b2e3d7fad1d50cbc580f899ca6c27d" exitCode=0 Jul 07 14:13:47.623524 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:47.623410 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" event={"ID":"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe","Type":"ContainerDied","Data":"478e717daaf58775d2cc3e2c35963d09b9b2e3d7fad1d50cbc580f899ca6c27d"} Jul 07 14:13:47.623524 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:47.623457 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" event={"ID":"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe","Type":"ContainerStarted","Data":"8f43ebc4dc815bacaba512f7be1c65a29005a7d492d6d1a0c571c37f3e285884"} Jul 07 14:13:48.340947 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:48.340876 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:48.644623 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:48.644538 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:48.928557 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:48.928474 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-98snv"] Jul 07 14:13:48.931003 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:48.930978 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-98snv" Jul 07 14:13:48.933624 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:48.933593 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 14:13:48.934342 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:48.934324 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xmfrb\"" Jul 07 14:13:48.934461 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:48.934378 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 14:13:48.938285 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:48.938242 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-98snv"] Jul 07 14:13:49.072504 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:49.072473 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3bb6944-67d3-4669-9caa-5e15e749aff5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-98snv\" (UID: \"c3bb6944-67d3-4669-9caa-5e15e749aff5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-98snv" Jul 07 14:13:49.072674 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:49.072568 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnpvq\" (UniqueName: \"kubernetes.io/projected/c3bb6944-67d3-4669-9caa-5e15e749aff5-kube-api-access-pnpvq\") pod \"cert-manager-cainjector-559cdd8c87-98snv\" (UID: \"c3bb6944-67d3-4669-9caa-5e15e749aff5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-98snv" Jul 07 14:13:49.173538 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:49.173500 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnpvq\" (UniqueName: \"kubernetes.io/projected/c3bb6944-67d3-4669-9caa-5e15e749aff5-kube-api-access-pnpvq\") pod \"cert-manager-cainjector-559cdd8c87-98snv\" (UID: \"c3bb6944-67d3-4669-9caa-5e15e749aff5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-98snv" Jul 07 14:13:49.173728 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:49.173572 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3bb6944-67d3-4669-9caa-5e15e749aff5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-98snv\" (UID: \"c3bb6944-67d3-4669-9caa-5e15e749aff5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-98snv" Jul 07 14:13:49.186492 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:49.186423 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3bb6944-67d3-4669-9caa-5e15e749aff5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-98snv\" (UID: \"c3bb6944-67d3-4669-9caa-5e15e749aff5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-98snv" Jul 07 14:13:49.186639 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:49.186580 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnpvq\" (UniqueName: \"kubernetes.io/projected/c3bb6944-67d3-4669-9caa-5e15e749aff5-kube-api-access-pnpvq\") pod \"cert-manager-cainjector-559cdd8c87-98snv\" (UID: \"c3bb6944-67d3-4669-9caa-5e15e749aff5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-98snv" Jul 07 14:13:49.251921 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:49.251883 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-98snv" Jul 07 14:13:49.834771 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:49.834741 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-98snv"] Jul 07 14:13:49.837443 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:13:49.837416 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3bb6944_67d3_4669_9caa_5e15e749aff5.slice/crio-0a0d1a74a74495767caf954b5be98a6d0b1a0241ed24aba84175c20888a8d8fb WatchSource:0}: Error finding container 0a0d1a74a74495767caf954b5be98a6d0b1a0241ed24aba84175c20888a8d8fb: Status 404 returned error can't find the container with id 0a0d1a74a74495767caf954b5be98a6d0b1a0241ed24aba84175c20888a8d8fb Jul 07 14:13:50.638650 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:50.638608 2560 generic.go:356] "Generic (PLEG): container finished" podID="dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe" containerID="0de659846e505b0472b6d350a691e4a50a76ddb13b7ba2d771356cd21f4fffb8" exitCode=0 Jul 07 14:13:50.638836 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:50.638700 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" event={"ID":"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe","Type":"ContainerDied","Data":"0de659846e505b0472b6d350a691e4a50a76ddb13b7ba2d771356cd21f4fffb8"} Jul 07 14:13:50.640309 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:50.640284 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-98snv" event={"ID":"c3bb6944-67d3-4669-9caa-5e15e749aff5","Type":"ContainerStarted","Data":"0a0d1a74a74495767caf954b5be98a6d0b1a0241ed24aba84175c20888a8d8fb"} Jul 07 14:13:50.994865 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:50.994755 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:51.049657 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:51.049615 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:51.648050 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:51.648018 2560 generic.go:356] "Generic (PLEG): container finished" podID="dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe" containerID="b478db1efb4767103be0a7549b21505bf098cde0f14a7546e426bbe0ce93f432" exitCode=0 Jul 07 14:13:51.648230 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:51.648108 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" event={"ID":"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe","Type":"ContainerDied","Data":"b478db1efb4767103be0a7549b21505bf098cde0f14a7546e426bbe0ce93f432"} Jul 07 14:13:52.789419 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:52.789388 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" Jul 07 14:13:52.908397 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:52.908303 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-util\") pod \"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe\" (UID: \"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe\") " Jul 07 14:13:52.908397 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:52.908383 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c24f\" (UniqueName: \"kubernetes.io/projected/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-kube-api-access-9c24f\") pod \"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe\" (UID: \"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe\") " Jul 07 14:13:52.908637 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:52.908446 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-bundle\") pod \"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe\" (UID: \"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe\") " Jul 07 14:13:52.908783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:52.908757 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-bundle" (OuterVolumeSpecName: "bundle") pod "dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe" (UID: "dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:13:52.910628 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:52.910602 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-kube-api-access-9c24f" (OuterVolumeSpecName: "kube-api-access-9c24f") pod "dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe" (UID: "dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe"). InnerVolumeSpecName "kube-api-access-9c24f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:13:52.915029 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:52.914990 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-util" (OuterVolumeSpecName: "util") pod "dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe" (UID: "dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:13:53.009291 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:53.009217 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-util\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:13:53.009291 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:53.009248 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c24f\" (UniqueName: \"kubernetes.io/projected/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-kube-api-access-9c24f\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:13:53.009291 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:53.009257 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe-bundle\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:13:53.658686 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:53.658652 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" Jul 07 14:13:53.658957 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:53.658688 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqj4d2" event={"ID":"dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe","Type":"ContainerDied","Data":"8f43ebc4dc815bacaba512f7be1c65a29005a7d492d6d1a0c571c37f3e285884"} Jul 07 14:13:53.658957 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:53.658746 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f43ebc4dc815bacaba512f7be1c65a29005a7d492d6d1a0c571c37f3e285884" Jul 07 14:13:53.660101 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:53.660074 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-98snv" event={"ID":"c3bb6944-67d3-4669-9caa-5e15e749aff5","Type":"ContainerStarted","Data":"967e263b36626f56c789979f99880c8676832ea00cfc878d1a4d82ecf6b000db"} Jul 07 14:13:53.676537 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:53.676496 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-98snv" podStartSLOduration=2.8675959779999998 podStartE2EDuration="5.676481945s" totalImagesPullingTime="2.808885967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:13:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:13:53.674876229 +0000 UTC m=+392.107118731" watchObservedRunningTime="2026-07-07 14:13:53.676481945 +0000 UTC m=+392.108724447" Jul 07 14:13:55.553243 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:55.553211 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:56.285249 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.285216 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2pb6j"] Jul 07 14:13:56.285526 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.285513 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe" containerName="util" Jul 07 14:13:56.285572 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.285528 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe" containerName="util" Jul 07 14:13:56.285572 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.285540 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe" containerName="pull" Jul 07 14:13:56.285572 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.285546 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe" containerName="pull" Jul 07 14:13:56.285572 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.285564 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe" containerName="extract" Jul 07 14:13:56.285572 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.285569 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe" containerName="extract" Jul 07 14:13:56.285718 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.285619 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd8e2b8a-ddd5-4dfc-ad86-c2664e372abe" containerName="extract" Jul 07 14:13:56.287709 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.287694 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2pb6j" Jul 07 14:13:56.293945 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.293920 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-cxvz7\"" Jul 07 14:13:56.298690 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.298669 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2pb6j"] Jul 07 14:13:56.439823 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.439766 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a06ab8d-db16-40d0-8ab7-caeeb2b657a0-bound-sa-token\") pod \"cert-manager-55b4d954d5-2pb6j\" (UID: \"6a06ab8d-db16-40d0-8ab7-caeeb2b657a0\") " pod="cert-manager/cert-manager-55b4d954d5-2pb6j" Jul 07 14:13:56.440001 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.439855 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxdtb\" (UniqueName: \"kubernetes.io/projected/6a06ab8d-db16-40d0-8ab7-caeeb2b657a0-kube-api-access-mxdtb\") pod \"cert-manager-55b4d954d5-2pb6j\" (UID: \"6a06ab8d-db16-40d0-8ab7-caeeb2b657a0\") " pod="cert-manager/cert-manager-55b4d954d5-2pb6j" Jul 07 14:13:56.540914 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.540827 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxdtb\" (UniqueName: \"kubernetes.io/projected/6a06ab8d-db16-40d0-8ab7-caeeb2b657a0-kube-api-access-mxdtb\") pod \"cert-manager-55b4d954d5-2pb6j\" (UID: \"6a06ab8d-db16-40d0-8ab7-caeeb2b657a0\") " pod="cert-manager/cert-manager-55b4d954d5-2pb6j" Jul 07 14:13:56.540914 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.540907 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a06ab8d-db16-40d0-8ab7-caeeb2b657a0-bound-sa-token\") pod \"cert-manager-55b4d954d5-2pb6j\" (UID: \"6a06ab8d-db16-40d0-8ab7-caeeb2b657a0\") " pod="cert-manager/cert-manager-55b4d954d5-2pb6j" Jul 07 14:13:56.549476 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.549445 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxdtb\" (UniqueName: \"kubernetes.io/projected/6a06ab8d-db16-40d0-8ab7-caeeb2b657a0-kube-api-access-mxdtb\") pod \"cert-manager-55b4d954d5-2pb6j\" (UID: \"6a06ab8d-db16-40d0-8ab7-caeeb2b657a0\") " pod="cert-manager/cert-manager-55b4d954d5-2pb6j" Jul 07 14:13:56.549592 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.549454 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6a06ab8d-db16-40d0-8ab7-caeeb2b657a0-bound-sa-token\") pod \"cert-manager-55b4d954d5-2pb6j\" (UID: \"6a06ab8d-db16-40d0-8ab7-caeeb2b657a0\") " pod="cert-manager/cert-manager-55b4d954d5-2pb6j" Jul 07 14:13:56.598549 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.598517 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2pb6j" Jul 07 14:13:56.731594 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:56.731562 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2pb6j"] Jul 07 14:13:56.734570 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:13:56.734544 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a06ab8d_db16_40d0_8ab7_caeeb2b657a0.slice/crio-35db1cf877c78ebc3fbcb2603f3f57bd7ab35c97189f18fa2cd816b5cdfb0d3e WatchSource:0}: Error finding container 35db1cf877c78ebc3fbcb2603f3f57bd7ab35c97189f18fa2cd816b5cdfb0d3e: Status 404 returned error can't find the container with id 35db1cf877c78ebc3fbcb2603f3f57bd7ab35c97189f18fa2cd816b5cdfb0d3e Jul 07 14:13:57.416800 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:57.416751 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:57.514565 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:57.514519 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:13:57.679713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:57.679630 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2pb6j" event={"ID":"6a06ab8d-db16-40d0-8ab7-caeeb2b657a0","Type":"ContainerStarted","Data":"d3c12c19bdb943356b6ffa89e863c30243507654a48f20198f12f5f02898e82f"} Jul 07 14:13:57.679713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:57.679678 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2pb6j" event={"ID":"6a06ab8d-db16-40d0-8ab7-caeeb2b657a0","Type":"ContainerStarted","Data":"35db1cf877c78ebc3fbcb2603f3f57bd7ab35c97189f18fa2cd816b5cdfb0d3e"} Jul 07 14:13:57.695363 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:13:57.695320 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-2pb6j" podStartSLOduration=1.695303922 podStartE2EDuration="1.695303922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:13:57.694498598 +0000 UTC m=+396.126741102" watchObservedRunningTime="2026-07-07 14:13:57.695303922 +0000 UTC m=+396.127546422" Jul 07 14:14:03.925465 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:03.925433 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:04.029555 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:04.029523 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:05.675492 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:05.675454 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:07.078358 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.078320 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k"] Jul 07 14:14:07.119387 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.119354 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k"] Jul 07 14:14:07.119560 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.119489 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" Jul 07 14:14:07.122341 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.122322 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5nmlx\"" Jul 07 14:14:07.122454 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.122324 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 14:14:07.122968 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.122945 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 14:14:07.130198 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.130168 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k\" (UID: \"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" Jul 07 14:14:07.130361 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.130317 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxqpk\" (UniqueName: \"kubernetes.io/projected/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-kube-api-access-gxqpk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k\" (UID: \"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" Jul 07 14:14:07.130421 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.130370 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k\" (UID: \"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" Jul 07 14:14:07.230716 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.230674 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k\" (UID: \"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" Jul 07 14:14:07.230904 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.230740 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxqpk\" (UniqueName: \"kubernetes.io/projected/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-kube-api-access-gxqpk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k\" (UID: \"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" Jul 07 14:14:07.230904 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.230791 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k\" (UID: \"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" Jul 07 14:14:07.231149 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.231126 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k\" (UID: \"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" Jul 07 14:14:07.231211 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.231142 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k\" (UID: \"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" Jul 07 14:14:07.239760 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.239734 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxqpk\" (UniqueName: \"kubernetes.io/projected/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-kube-api-access-gxqpk\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k\" (UID: \"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" Jul 07 14:14:07.434669 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.434585 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" Jul 07 14:14:07.584006 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.583979 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k"] Jul 07 14:14:07.586104 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:14:07.586077 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5961c6_8a13_47fa_9b1f_7eea7a7c1487.slice/crio-ad5fc781e67c2e2033745a7cee049222dc11daec3af185d11fcbd448c9cc887f WatchSource:0}: Error finding container ad5fc781e67c2e2033745a7cee049222dc11daec3af185d11fcbd448c9cc887f: Status 404 returned error can't find the container with id ad5fc781e67c2e2033745a7cee049222dc11daec3af185d11fcbd448c9cc887f Jul 07 14:14:07.724160 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.724131 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" event={"ID":"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487","Type":"ContainerStarted","Data":"c11df0f61606c1093ff3f9bbfc0ce8b337eae4ab40b8f16f22105e1dbc327a04"} Jul 07 14:14:07.724319 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:07.724191 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" event={"ID":"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487","Type":"ContainerStarted","Data":"ad5fc781e67c2e2033745a7cee049222dc11daec3af185d11fcbd448c9cc887f"} Jul 07 14:14:08.730652 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:08.730617 2560 generic.go:356] "Generic (PLEG): container finished" podID="5c5961c6-8a13-47fa-9b1f-7eea7a7c1487" containerID="c11df0f61606c1093ff3f9bbfc0ce8b337eae4ab40b8f16f22105e1dbc327a04" exitCode=0 Jul 07 14:14:08.730652 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:08.730649 2560 generic.go:356] "Generic (PLEG): container finished" podID="5c5961c6-8a13-47fa-9b1f-7eea7a7c1487" containerID="4c4ce4ec0e2978989dec1ba38cd8c51be613e7d817c9ab72310872cec09b8482" exitCode=0 Jul 07 14:14:08.731067 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:08.730697 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" event={"ID":"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487","Type":"ContainerDied","Data":"c11df0f61606c1093ff3f9bbfc0ce8b337eae4ab40b8f16f22105e1dbc327a04"} Jul 07 14:14:08.731067 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:08.730752 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" event={"ID":"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487","Type":"ContainerDied","Data":"4c4ce4ec0e2978989dec1ba38cd8c51be613e7d817c9ab72310872cec09b8482"} Jul 07 14:14:09.737136 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:09.737102 2560 generic.go:356] "Generic (PLEG): container finished" podID="5c5961c6-8a13-47fa-9b1f-7eea7a7c1487" containerID="1dcc2d86c2e2012779d467f0da98602ab9ba2f1a50de6d0529e2b25f14eca83f" exitCode=0 Jul 07 14:14:09.737602 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:09.737172 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" event={"ID":"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487","Type":"ContainerDied","Data":"1dcc2d86c2e2012779d467f0da98602ab9ba2f1a50de6d0529e2b25f14eca83f"} Jul 07 14:14:10.767651 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:10.767623 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:10.867941 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:10.867918 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:10.875176 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:10.875155 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" Jul 07 14:14:10.956232 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:10.956200 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-bundle\") pod \"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487\" (UID: \"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487\") " Jul 07 14:14:10.956413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:10.956334 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-util\") pod \"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487\" (UID: \"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487\") " Jul 07 14:14:10.956413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:10.956386 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxqpk\" (UniqueName: \"kubernetes.io/projected/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-kube-api-access-gxqpk\") pod \"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487\" (UID: \"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487\") " Jul 07 14:14:10.956991 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:10.956960 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-bundle" (OuterVolumeSpecName: "bundle") pod "5c5961c6-8a13-47fa-9b1f-7eea7a7c1487" (UID: "5c5961c6-8a13-47fa-9b1f-7eea7a7c1487"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:14:10.958657 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:10.958633 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-kube-api-access-gxqpk" (OuterVolumeSpecName: "kube-api-access-gxqpk") pod "5c5961c6-8a13-47fa-9b1f-7eea7a7c1487" (UID: "5c5961c6-8a13-47fa-9b1f-7eea7a7c1487"). InnerVolumeSpecName "kube-api-access-gxqpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:14:10.962155 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:10.962116 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-util" (OuterVolumeSpecName: "util") pod "5c5961c6-8a13-47fa-9b1f-7eea7a7c1487" (UID: "5c5961c6-8a13-47fa-9b1f-7eea7a7c1487"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:14:11.056951 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:11.056922 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-util\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:14:11.056951 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:11.056951 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxqpk\" (UniqueName: \"kubernetes.io/projected/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-kube-api-access-gxqpk\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:14:11.057126 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:11.056961 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c5961c6-8a13-47fa-9b1f-7eea7a7c1487-bundle\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:14:11.748072 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:11.748041 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" event={"ID":"5c5961c6-8a13-47fa-9b1f-7eea7a7c1487","Type":"ContainerDied","Data":"ad5fc781e67c2e2033745a7cee049222dc11daec3af185d11fcbd448c9cc887f"} Jul 07 14:14:11.748233 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:11.748082 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad5fc781e67c2e2033745a7cee049222dc11daec3af185d11fcbd448c9cc887f" Jul 07 14:14:11.748233 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:11.748093 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcvg7k" Jul 07 14:14:18.592593 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:18.592562 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:18.692427 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:18.692386 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:19.218202 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.218173 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:19.368872 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.368842 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:19.476912 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.476827 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:19.512036 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.511998 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5"] Jul 07 14:14:19.512399 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.512383 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c5961c6-8a13-47fa-9b1f-7eea7a7c1487" containerName="util" Jul 07 14:14:19.512399 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.512400 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c5961c6-8a13-47fa-9b1f-7eea7a7c1487" containerName="util" Jul 07 14:14:19.512543 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.512415 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c5961c6-8a13-47fa-9b1f-7eea7a7c1487" containerName="pull" Jul 07 14:14:19.512543 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.512422 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c5961c6-8a13-47fa-9b1f-7eea7a7c1487" containerName="pull" Jul 07 14:14:19.512543 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.512430 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c5961c6-8a13-47fa-9b1f-7eea7a7c1487" containerName="extract" Jul 07 14:14:19.512543 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.512438 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c5961c6-8a13-47fa-9b1f-7eea7a7c1487" containerName="extract" Jul 07 14:14:19.512543 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.512535 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c5961c6-8a13-47fa-9b1f-7eea7a7c1487" containerName="extract" Jul 07 14:14:19.514836 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.514817 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.520822 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.520794 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 14:14:19.523604 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.523584 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-55ddl\"" Jul 07 14:14:19.523725 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.523630 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 14:14:19.523725 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.523660 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 14:14:19.523725 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.523584 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 14:14:19.533988 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.533967 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5"] Jul 07 14:14:19.624696 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.624655 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cc32b38-d167-4ef5-acef-19749b6f76a6-tmp\") pod \"opendatahub-operator-controller-manager-55dcd65d78-5xcq5\" (UID: \"6cc32b38-d167-4ef5-acef-19749b6f76a6\") " pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.624696 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.624703 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6cc32b38-d167-4ef5-acef-19749b6f76a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-55dcd65d78-5xcq5\" (UID: \"6cc32b38-d167-4ef5-acef-19749b6f76a6\") " pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.625127 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.624722 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6cc32b38-d167-4ef5-acef-19749b6f76a6-manifests\") pod \"opendatahub-operator-controller-manager-55dcd65d78-5xcq5\" (UID: \"6cc32b38-d167-4ef5-acef-19749b6f76a6\") " pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.625127 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.624796 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z69vd\" (UniqueName: \"kubernetes.io/projected/6cc32b38-d167-4ef5-acef-19749b6f76a6-kube-api-access-z69vd\") pod \"opendatahub-operator-controller-manager-55dcd65d78-5xcq5\" (UID: \"6cc32b38-d167-4ef5-acef-19749b6f76a6\") " pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.625127 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.624830 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6cc32b38-d167-4ef5-acef-19749b6f76a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55dcd65d78-5xcq5\" (UID: \"6cc32b38-d167-4ef5-acef-19749b6f76a6\") " pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.683453 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.683415 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:19.725173 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.725136 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cc32b38-d167-4ef5-acef-19749b6f76a6-tmp\") pod \"opendatahub-operator-controller-manager-55dcd65d78-5xcq5\" (UID: \"6cc32b38-d167-4ef5-acef-19749b6f76a6\") " pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.725358 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.725184 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6cc32b38-d167-4ef5-acef-19749b6f76a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-55dcd65d78-5xcq5\" (UID: \"6cc32b38-d167-4ef5-acef-19749b6f76a6\") " pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.725358 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.725215 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6cc32b38-d167-4ef5-acef-19749b6f76a6-manifests\") pod \"opendatahub-operator-controller-manager-55dcd65d78-5xcq5\" (UID: \"6cc32b38-d167-4ef5-acef-19749b6f76a6\") " pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.725358 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.725246 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z69vd\" (UniqueName: \"kubernetes.io/projected/6cc32b38-d167-4ef5-acef-19749b6f76a6-kube-api-access-z69vd\") pod \"opendatahub-operator-controller-manager-55dcd65d78-5xcq5\" (UID: \"6cc32b38-d167-4ef5-acef-19749b6f76a6\") " pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.725358 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.725298 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6cc32b38-d167-4ef5-acef-19749b6f76a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55dcd65d78-5xcq5\" (UID: \"6cc32b38-d167-4ef5-acef-19749b6f76a6\") " pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.725672 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.725645 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6cc32b38-d167-4ef5-acef-19749b6f76a6-manifests\") pod \"opendatahub-operator-controller-manager-55dcd65d78-5xcq5\" (UID: \"6cc32b38-d167-4ef5-acef-19749b6f76a6\") " pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.727693 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.727635 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cc32b38-d167-4ef5-acef-19749b6f76a6-tmp\") pod \"opendatahub-operator-controller-manager-55dcd65d78-5xcq5\" (UID: \"6cc32b38-d167-4ef5-acef-19749b6f76a6\") " pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.727831 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.727807 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6cc32b38-d167-4ef5-acef-19749b6f76a6-webhook-cert\") pod \"opendatahub-operator-controller-manager-55dcd65d78-5xcq5\" (UID: \"6cc32b38-d167-4ef5-acef-19749b6f76a6\") " pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.727913 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.727816 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6cc32b38-d167-4ef5-acef-19749b6f76a6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55dcd65d78-5xcq5\" (UID: \"6cc32b38-d167-4ef5-acef-19749b6f76a6\") " pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.735618 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.735597 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z69vd\" (UniqueName: \"kubernetes.io/projected/6cc32b38-d167-4ef5-acef-19749b6f76a6-kube-api-access-z69vd\") pod \"opendatahub-operator-controller-manager-55dcd65d78-5xcq5\" (UID: \"6cc32b38-d167-4ef5-acef-19749b6f76a6\") " pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.783904 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.783868 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:19.830443 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.830417 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:19.885788 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.885754 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:19.985172 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.985147 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5"] Jul 07 14:14:19.986169 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:19.986145 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:19.987289 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:14:19.987250 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc32b38_d167_4ef5_acef_19749b6f76a6.slice/crio-afa4ea2cd164d158e35beb4eaa18d0bc0ff6c63306fb1f61b7c1b7af97bd5a1c WatchSource:0}: Error finding container afa4ea2cd164d158e35beb4eaa18d0bc0ff6c63306fb1f61b7c1b7af97bd5a1c: Status 404 returned error can't find the container with id afa4ea2cd164d158e35beb4eaa18d0bc0ff6c63306fb1f61b7c1b7af97bd5a1c Jul 07 14:14:20.045016 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.044984 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt"] Jul 07 14:14:20.047840 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.047824 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:20.052623 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.052585 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 14:14:20.052623 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.052623 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4tmln\"" Jul 07 14:14:20.052828 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.052591 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 14:14:20.052828 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.052591 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 14:14:20.052828 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.052743 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:14:20.052949 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.052939 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 14:14:20.063323 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.063300 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt"] Jul 07 14:14:20.109726 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.109692 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:20.130050 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.130023 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f446dfd7-14e6-46c4-929b-adc563464d1d-manager-config\") pod \"lws-controller-manager-6489488d88-4dbdt\" (UID: \"f446dfd7-14e6-46c4-929b-adc563464d1d\") " pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:20.130205 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.130083 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnx76\" (UniqueName: \"kubernetes.io/projected/f446dfd7-14e6-46c4-929b-adc563464d1d-kube-api-access-pnx76\") pod \"lws-controller-manager-6489488d88-4dbdt\" (UID: \"f446dfd7-14e6-46c4-929b-adc563464d1d\") " pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:20.130205 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.130127 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f446dfd7-14e6-46c4-929b-adc563464d1d-metrics-cert\") pod \"lws-controller-manager-6489488d88-4dbdt\" (UID: \"f446dfd7-14e6-46c4-929b-adc563464d1d\") " pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:20.130205 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.130157 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f446dfd7-14e6-46c4-929b-adc563464d1d-cert\") pod \"lws-controller-manager-6489488d88-4dbdt\" (UID: \"f446dfd7-14e6-46c4-929b-adc563464d1d\") " pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:20.231181 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.231153 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f446dfd7-14e6-46c4-929b-adc563464d1d-metrics-cert\") pod \"lws-controller-manager-6489488d88-4dbdt\" (UID: \"f446dfd7-14e6-46c4-929b-adc563464d1d\") " pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:20.231181 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.231194 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f446dfd7-14e6-46c4-929b-adc563464d1d-cert\") pod \"lws-controller-manager-6489488d88-4dbdt\" (UID: \"f446dfd7-14e6-46c4-929b-adc563464d1d\") " pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:20.231448 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.231224 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f446dfd7-14e6-46c4-929b-adc563464d1d-manager-config\") pod \"lws-controller-manager-6489488d88-4dbdt\" (UID: \"f446dfd7-14e6-46c4-929b-adc563464d1d\") " pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:20.231448 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.231284 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnx76\" (UniqueName: \"kubernetes.io/projected/f446dfd7-14e6-46c4-929b-adc563464d1d-kube-api-access-pnx76\") pod \"lws-controller-manager-6489488d88-4dbdt\" (UID: \"f446dfd7-14e6-46c4-929b-adc563464d1d\") " pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:20.232053 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.231983 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f446dfd7-14e6-46c4-929b-adc563464d1d-manager-config\") pod \"lws-controller-manager-6489488d88-4dbdt\" (UID: \"f446dfd7-14e6-46c4-929b-adc563464d1d\") " pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:20.233789 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.233768 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f446dfd7-14e6-46c4-929b-adc563464d1d-cert\") pod \"lws-controller-manager-6489488d88-4dbdt\" (UID: \"f446dfd7-14e6-46c4-929b-adc563464d1d\") " pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:20.233917 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.233895 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f446dfd7-14e6-46c4-929b-adc563464d1d-metrics-cert\") pod \"lws-controller-manager-6489488d88-4dbdt\" (UID: \"f446dfd7-14e6-46c4-929b-adc563464d1d\") " pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:20.239873 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.239813 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnx76\" (UniqueName: \"kubernetes.io/projected/f446dfd7-14e6-46c4-929b-adc563464d1d-kube-api-access-pnx76\") pod \"lws-controller-manager-6489488d88-4dbdt\" (UID: \"f446dfd7-14e6-46c4-929b-adc563464d1d\") " pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:20.358245 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.358206 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:20.523190 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.523161 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt"] Jul 07 14:14:20.525911 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:14:20.525873 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf446dfd7_14e6_46c4_929b_adc563464d1d.slice/crio-02bfe9058159342ea562a992629681075e82c5d2d83e5518ef7220794a6a5889 WatchSource:0}: Error finding container 02bfe9058159342ea562a992629681075e82c5d2d83e5518ef7220794a6a5889: Status 404 returned error can't find the container with id 02bfe9058159342ea562a992629681075e82c5d2d83e5518ef7220794a6a5889 Jul 07 14:14:20.789672 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.789581 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" event={"ID":"f446dfd7-14e6-46c4-929b-adc563464d1d","Type":"ContainerStarted","Data":"02bfe9058159342ea562a992629681075e82c5d2d83e5518ef7220794a6a5889"} Jul 07 14:14:20.790860 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:20.790838 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" event={"ID":"6cc32b38-d167-4ef5-acef-19749b6f76a6","Type":"ContainerStarted","Data":"afa4ea2cd164d158e35beb4eaa18d0bc0ff6c63306fb1f61b7c1b7af97bd5a1c"} Jul 07 14:14:23.809636 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:23.809588 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" event={"ID":"6cc32b38-d167-4ef5-acef-19749b6f76a6","Type":"ContainerStarted","Data":"b054bcb33929ebf7645e683a1f1fcb4fb9eff7a07135bbe48da88de1257e7ca7"} Jul 07 14:14:23.811079 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:23.811057 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" event={"ID":"f446dfd7-14e6-46c4-929b-adc563464d1d","Type":"ContainerStarted","Data":"cebc9a25d70a7d2dfa14b75fb699902637349f4a9c80927cd0d7904a5a51068c"} Jul 07 14:14:23.811201 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:23.811189 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:23.860979 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:23.860919 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" podStartSLOduration=0.896468894 podStartE2EDuration="3.860899527s" totalImagesPullingTime="2.964430633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:14:23.860132722 +0000 UTC m=+422.292375227" watchObservedRunningTime="2026-07-07 14:14:23.860899527 +0000 UTC m=+422.293142030" Jul 07 14:14:24.817109 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:24.817074 2560 generic.go:356] "Generic (PLEG): container finished" podID="6cc32b38-d167-4ef5-acef-19749b6f76a6" containerID="b054bcb33929ebf7645e683a1f1fcb4fb9eff7a07135bbe48da88de1257e7ca7" exitCode=0 Jul 07 14:14:24.817509 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:24.817151 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" event={"ID":"6cc32b38-d167-4ef5-acef-19749b6f76a6","Type":"ContainerDied","Data":"b054bcb33929ebf7645e683a1f1fcb4fb9eff7a07135bbe48da88de1257e7ca7"} Jul 07 14:14:25.823653 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:25.823620 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" event={"ID":"6cc32b38-d167-4ef5-acef-19749b6f76a6","Type":"ContainerStarted","Data":"394f1cbbd2f7baefdc638c79cdd99ef84699ae8976e4e74b9dab79488d587bc8"} Jul 07 14:14:25.824035 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:25.823752 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:25.845706 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:25.845659 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" podStartSLOduration=1.846072953 podStartE2EDuration="6.84564367s" totalImagesPullingTime="3.978194985s" totalInitContainerRuntime="1.021375732s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:14:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:14:25.844863603 +0000 UTC m=+424.277106123" watchObservedRunningTime="2026-07-07 14:14:25.84564367 +0000 UTC m=+424.277886172" Jul 07 14:14:27.518013 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:27.517948 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:27.581952 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:27.581915 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:29.259441 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:29.259395 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:29.360744 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:29.360703 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:30.897142 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:30.897103 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:30.991821 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:30.991787 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:34.819065 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:34.819035 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6489488d88-4dbdt" Jul 07 14:14:36.535708 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:36.535674 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:36.595973 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:36.595940 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:36.831151 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:36.831119 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55dcd65d78-5xcq5" Jul 07 14:14:38.878941 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:38.878913 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:38.990897 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:38.990868 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:40.151602 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:40.151573 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:40.239664 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:40.239636 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:40.458682 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:40.458572 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:40.626499 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:40.626457 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:40.742623 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:40.742500 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:40.864916 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:40.864887 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:41.462084 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:41.462050 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:41.522897 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:41.522865 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:42.416360 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:42.416327 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:48.447253 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.447220 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf"] Jul 07 14:14:48.450498 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.450479 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.453820 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.453798 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 14:14:48.453951 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.453798 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 14:14:48.453951 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.453872 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 14:14:48.454613 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.454597 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 14:14:48.454832 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.454818 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nnwlf\"" Jul 07 14:14:48.457478 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.457116 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 14:14:48.457637 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.457608 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 14:14:48.478366 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.478338 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf"] Jul 07 14:14:48.578914 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.578888 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.578914 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.578918 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wvlb\" (UniqueName: \"kubernetes.io/projected/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-kube-api-access-7wvlb\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.579151 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.578939 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.579151 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.578994 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.579151 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.579064 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.579151 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.579111 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.579337 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.579171 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.603759 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.603736 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:48.679985 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.679962 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.680095 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.680002 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.680095 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.680034 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.680180 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.680149 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wvlb\" (UniqueName: \"kubernetes.io/projected/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-kube-api-access-7wvlb\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.680230 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.680188 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.680313 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.680225 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.680371 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.680312 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.680662 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.680632 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.682564 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.682545 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.682674 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.682606 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.682736 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.682680 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.682736 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.682707 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.689712 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.689692 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wvlb\" (UniqueName: \"kubernetes.io/projected/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-kube-api-access-7wvlb\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.690513 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.690480 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e49ddb-f2e2-448b-a618-cd3cb935cb7c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-s69mf\" (UID: \"08e49ddb-f2e2-448b-a618-cd3cb935cb7c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.703066 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.703017 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:48.760240 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.760212 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:48.902045 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.902019 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf"] Jul 07 14:14:48.904186 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:14:48.904161 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e49ddb_f2e2_448b_a618_cd3cb935cb7c.slice/crio-7694f0d7e7635d731fcd0d734ace465202c7491f2dae40e0751a6665901e3582 WatchSource:0}: Error finding container 7694f0d7e7635d731fcd0d734ace465202c7491f2dae40e0751a6665901e3582: Status 404 returned error can't find the container with id 7694f0d7e7635d731fcd0d734ace465202c7491f2dae40e0751a6665901e3582 Jul 07 14:14:48.938023 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:48.937990 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" event={"ID":"08e49ddb-f2e2-448b-a618-cd3cb935cb7c","Type":"ContainerStarted","Data":"7694f0d7e7635d731fcd0d734ace465202c7491f2dae40e0751a6665901e3582"} Jul 07 14:14:49.485782 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:49.485750 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:49.746678 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:49.746603 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:50.225293 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:50.225241 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:51.126924 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:51.126882 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 14:14:51.127313 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:51.126964 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 07 14:14:51.325066 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:51.325037 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:51.422489 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:51.422376 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:51.494453 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:51.494392 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:51.680561 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:51.680469 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:51.956610 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:51.956520 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" event={"ID":"08e49ddb-f2e2-448b-a618-cd3cb935cb7c","Type":"ContainerStarted","Data":"f14751e06c2c6f0c5cd2978d46bc98c83050375cd6613cd2bec4e1b4d0f71f3e"} Jul 07 14:14:51.956781 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:51.956660 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:51.979081 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:51.979031 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" podStartSLOduration=1.758488526 podStartE2EDuration="3.979015975s" totalImagesPullingTime="2.220527449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:14:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:14:51.976666137 +0000 UTC m=+450.408908639" watchObservedRunningTime="2026-07-07 14:14:51.979015975 +0000 UTC m=+450.411258480" Jul 07 14:14:52.283880 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:52.283790 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:52.657699 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:52.657668 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:52.752685 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:52.752653 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:52.853080 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:52.853045 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:52.962918 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:52.962832 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-s69mf" Jul 07 14:14:53.125157 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:53.125110 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:53.231565 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:53.231482 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:14:57.245896 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:14:57.245863 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:15:03.020331 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:03.020293 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:15:03.073468 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:03.073441 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:15:19.970589 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:19.970558 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:15:20.074063 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:20.074032 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:15:40.407920 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:40.407883 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:15:40.507155 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:40.507107 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:15:40.610512 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:40.610480 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:15:55.908251 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:55.908216 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9"] Jul 07 14:15:55.912034 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:55.912017 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" Jul 07 14:15:55.914583 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:55.914557 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 14:15:55.914583 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:55.914577 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k7vsj\"" Jul 07 14:15:55.915470 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:55.915445 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 14:15:55.921249 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:55.921226 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9"] Jul 07 14:15:55.966668 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:55.966637 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9\" (UID: \"9c235c71-4d6c-46ef-a8c5-252a2d6e1480\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" Jul 07 14:15:55.966668 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:55.966670 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9\" (UID: \"9c235c71-4d6c-46ef-a8c5-252a2d6e1480\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" Jul 07 14:15:55.966882 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:55.966716 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4glq\" (UniqueName: \"kubernetes.io/projected/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-kube-api-access-w4glq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9\" (UID: \"9c235c71-4d6c-46ef-a8c5-252a2d6e1480\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" Jul 07 14:15:56.067482 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.067443 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9\" (UID: \"9c235c71-4d6c-46ef-a8c5-252a2d6e1480\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" Jul 07 14:15:56.067482 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.067484 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9\" (UID: \"9c235c71-4d6c-46ef-a8c5-252a2d6e1480\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" Jul 07 14:15:56.067673 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.067525 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4glq\" (UniqueName: \"kubernetes.io/projected/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-kube-api-access-w4glq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9\" (UID: \"9c235c71-4d6c-46ef-a8c5-252a2d6e1480\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" Jul 07 14:15:56.067841 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.067820 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9\" (UID: \"9c235c71-4d6c-46ef-a8c5-252a2d6e1480\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" Jul 07 14:15:56.067909 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.067890 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9\" (UID: \"9c235c71-4d6c-46ef-a8c5-252a2d6e1480\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" Jul 07 14:15:56.075858 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.075832 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4glq\" (UniqueName: \"kubernetes.io/projected/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-kube-api-access-w4glq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9\" (UID: \"9c235c71-4d6c-46ef-a8c5-252a2d6e1480\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" Jul 07 14:15:56.221817 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.221745 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" Jul 07 14:15:56.354555 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.354530 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9"] Jul 07 14:15:56.356812 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:15:56.356785 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c235c71_4d6c_46ef_a8c5_252a2d6e1480.slice/crio-f2ab1bc9e68aeb363a13dcbf7a117d07683c749c1064547883f9bd6a56cd23a6 WatchSource:0}: Error finding container f2ab1bc9e68aeb363a13dcbf7a117d07683c749c1064547883f9bd6a56cd23a6: Status 404 returned error can't find the container with id f2ab1bc9e68aeb363a13dcbf7a117d07683c749c1064547883f9bd6a56cd23a6 Jul 07 14:15:56.706246 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.706209 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2"] Jul 07 14:15:56.709608 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.709593 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" Jul 07 14:15:56.719635 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.719613 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2"] Jul 07 14:15:56.774521 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.774486 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2\" (UID: \"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" Jul 07 14:15:56.774521 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.774526 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2\" (UID: \"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" Jul 07 14:15:56.774752 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.774568 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q7d7\" (UniqueName: \"kubernetes.io/projected/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-kube-api-access-5q7d7\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2\" (UID: \"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" Jul 07 14:15:56.875713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.875677 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2\" (UID: \"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" Jul 07 14:15:56.875713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.875719 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2\" (UID: \"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" Jul 07 14:15:56.875974 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.875764 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q7d7\" (UniqueName: \"kubernetes.io/projected/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-kube-api-access-5q7d7\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2\" (UID: \"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" Jul 07 14:15:56.876134 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.876113 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2\" (UID: \"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" Jul 07 14:15:56.876195 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.876130 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2\" (UID: \"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" Jul 07 14:15:56.884247 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:56.884229 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q7d7\" (UniqueName: \"kubernetes.io/projected/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-kube-api-access-5q7d7\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2\" (UID: \"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" Jul 07 14:15:57.019878 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.019780 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" Jul 07 14:15:57.151859 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.151755 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2"] Jul 07 14:15:57.154659 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:15:57.154638 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d8d24f_9fb2_40f7_be6d_aeb41a5f5669.slice/crio-41035837bce736d0f8089d63c0ccc02f444c1bf69f09e8de286c8f80daab5f8d WatchSource:0}: Error finding container 41035837bce736d0f8089d63c0ccc02f444c1bf69f09e8de286c8f80daab5f8d: Status 404 returned error can't find the container with id 41035837bce736d0f8089d63c0ccc02f444c1bf69f09e8de286c8f80daab5f8d Jul 07 14:15:57.260221 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.260192 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" event={"ID":"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669","Type":"ContainerStarted","Data":"e256c9b7571dd6e0ab1351a55b5fb7efb7f4ad73881e0ee439d2d2f8ce21c2ee"} Jul 07 14:15:57.260384 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.260251 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" event={"ID":"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669","Type":"ContainerStarted","Data":"41035837bce736d0f8089d63c0ccc02f444c1bf69f09e8de286c8f80daab5f8d"} Jul 07 14:15:57.261994 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.261952 2560 generic.go:356] "Generic (PLEG): container finished" podID="9c235c71-4d6c-46ef-a8c5-252a2d6e1480" containerID="e7ec6310ce91ce6f7c9fc0549ae52be3ab47b4fd9555725dec2d4be09d2b497e" exitCode=0 Jul 07 14:15:57.261994 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.261987 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" event={"ID":"9c235c71-4d6c-46ef-a8c5-252a2d6e1480","Type":"ContainerDied","Data":"e7ec6310ce91ce6f7c9fc0549ae52be3ab47b4fd9555725dec2d4be09d2b497e"} Jul 07 14:15:57.262153 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.262020 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" event={"ID":"9c235c71-4d6c-46ef-a8c5-252a2d6e1480","Type":"ContainerStarted","Data":"f2ab1bc9e68aeb363a13dcbf7a117d07683c749c1064547883f9bd6a56cd23a6"} Jul 07 14:15:57.323365 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.323328 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf"] Jul 07 14:15:57.326821 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.326798 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" Jul 07 14:15:57.327596 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.327572 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf"] Jul 07 14:15:57.380285 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.380240 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q2sr\" (UniqueName: \"kubernetes.io/projected/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-kube-api-access-7q2sr\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf\" (UID: \"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" Jul 07 14:15:57.380456 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.380353 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf\" (UID: \"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" Jul 07 14:15:57.380456 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.380439 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf\" (UID: \"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" Jul 07 14:15:57.480979 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.480951 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf\" (UID: \"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" Jul 07 14:15:57.481164 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.480988 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q2sr\" (UniqueName: \"kubernetes.io/projected/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-kube-api-access-7q2sr\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf\" (UID: \"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" Jul 07 14:15:57.481164 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.481131 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf\" (UID: \"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" Jul 07 14:15:57.481444 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.481421 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf\" (UID: \"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" Jul 07 14:15:57.481516 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.481445 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf\" (UID: \"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" Jul 07 14:15:57.489189 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.489168 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q2sr\" (UniqueName: \"kubernetes.io/projected/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-kube-api-access-7q2sr\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf\" (UID: \"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" Jul 07 14:15:57.636427 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.636343 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" Jul 07 14:15:57.766783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.766754 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf"] Jul 07 14:15:57.768753 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:15:57.768727 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6dd2e1_cbbc_44f5_83bb_c1892c2e067f.slice/crio-b06a34327e0fb7da8d1242829dac28b94dcdd3740762bcd49fb22a6a77ecb3d2 WatchSource:0}: Error finding container b06a34327e0fb7da8d1242829dac28b94dcdd3740762bcd49fb22a6a77ecb3d2: Status 404 returned error can't find the container with id b06a34327e0fb7da8d1242829dac28b94dcdd3740762bcd49fb22a6a77ecb3d2 Jul 07 14:15:57.915675 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.915606 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj"] Jul 07 14:15:57.919021 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.919006 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" Jul 07 14:15:57.927502 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.927477 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj"] Jul 07 14:15:57.985843 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.985809 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a529a61-fb46-46dc-be67-869a613ea05f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj\" (UID: \"2a529a61-fb46-46dc-be67-869a613ea05f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" Jul 07 14:15:57.986036 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.985858 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a529a61-fb46-46dc-be67-869a613ea05f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj\" (UID: \"2a529a61-fb46-46dc-be67-869a613ea05f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" Jul 07 14:15:57.986036 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:57.985959 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmbjr\" (UniqueName: \"kubernetes.io/projected/2a529a61-fb46-46dc-be67-869a613ea05f-kube-api-access-hmbjr\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj\" (UID: \"2a529a61-fb46-46dc-be67-869a613ea05f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" Jul 07 14:15:58.086534 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:58.086504 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbjr\" (UniqueName: \"kubernetes.io/projected/2a529a61-fb46-46dc-be67-869a613ea05f-kube-api-access-hmbjr\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj\" (UID: \"2a529a61-fb46-46dc-be67-869a613ea05f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" Jul 07 14:15:58.086885 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:58.086565 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a529a61-fb46-46dc-be67-869a613ea05f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj\" (UID: \"2a529a61-fb46-46dc-be67-869a613ea05f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" Jul 07 14:15:58.086885 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:58.086598 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a529a61-fb46-46dc-be67-869a613ea05f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj\" (UID: \"2a529a61-fb46-46dc-be67-869a613ea05f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" Jul 07 14:15:58.086959 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:58.086926 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a529a61-fb46-46dc-be67-869a613ea05f-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj\" (UID: \"2a529a61-fb46-46dc-be67-869a613ea05f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" Jul 07 14:15:58.086994 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:58.086959 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a529a61-fb46-46dc-be67-869a613ea05f-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj\" (UID: \"2a529a61-fb46-46dc-be67-869a613ea05f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" Jul 07 14:15:58.094808 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:58.094786 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmbjr\" (UniqueName: \"kubernetes.io/projected/2a529a61-fb46-46dc-be67-869a613ea05f-kube-api-access-hmbjr\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj\" (UID: \"2a529a61-fb46-46dc-be67-869a613ea05f\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" Jul 07 14:15:58.228982 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:58.228900 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" Jul 07 14:15:58.270220 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:58.270186 2560 generic.go:356] "Generic (PLEG): container finished" podID="bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f" containerID="62122305800a8b94b1d8511505d5d91385b3b968d71f13b1951e3093b7f63d6e" exitCode=0 Jul 07 14:15:58.270432 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:58.270256 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" event={"ID":"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f","Type":"ContainerDied","Data":"62122305800a8b94b1d8511505d5d91385b3b968d71f13b1951e3093b7f63d6e"} Jul 07 14:15:58.270432 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:58.270341 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" event={"ID":"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f","Type":"ContainerStarted","Data":"b06a34327e0fb7da8d1242829dac28b94dcdd3740762bcd49fb22a6a77ecb3d2"} Jul 07 14:15:58.272043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:58.272018 2560 generic.go:356] "Generic (PLEG): container finished" podID="f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669" containerID="e256c9b7571dd6e0ab1351a55b5fb7efb7f4ad73881e0ee439d2d2f8ce21c2ee" exitCode=0 Jul 07 14:15:58.272227 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:58.272075 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" event={"ID":"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669","Type":"ContainerDied","Data":"e256c9b7571dd6e0ab1351a55b5fb7efb7f4ad73881e0ee439d2d2f8ce21c2ee"} Jul 07 14:15:58.369551 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:58.369507 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj"] Jul 07 14:15:58.372756 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:15:58.372731 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a529a61_fb46_46dc_be67_869a613ea05f.slice/crio-9175bc2fb736dc59d4f0e4f36a6c7a1f29eec14d5f1f1a67b1cbdebb523609ee WatchSource:0}: Error finding container 9175bc2fb736dc59d4f0e4f36a6c7a1f29eec14d5f1f1a67b1cbdebb523609ee: Status 404 returned error can't find the container with id 9175bc2fb736dc59d4f0e4f36a6c7a1f29eec14d5f1f1a67b1cbdebb523609ee Jul 07 14:15:59.279329 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:59.279292 2560 generic.go:356] "Generic (PLEG): container finished" podID="9c235c71-4d6c-46ef-a8c5-252a2d6e1480" containerID="ca23eadcd5cc6d5bbd3deb981ec9c08d006ec7b884c9f263b207f49b4e5a96e1" exitCode=0 Jul 07 14:15:59.279742 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:59.279376 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" event={"ID":"9c235c71-4d6c-46ef-a8c5-252a2d6e1480","Type":"ContainerDied","Data":"ca23eadcd5cc6d5bbd3deb981ec9c08d006ec7b884c9f263b207f49b4e5a96e1"} Jul 07 14:15:59.280942 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:59.280921 2560 generic.go:356] "Generic (PLEG): container finished" podID="2a529a61-fb46-46dc-be67-869a613ea05f" containerID="a71aa1888695b076e2caf7f7148ebc8b4abcc0366209531e7da39f77aac46959" exitCode=0 Jul 07 14:15:59.281046 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:59.280952 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" event={"ID":"2a529a61-fb46-46dc-be67-869a613ea05f","Type":"ContainerDied","Data":"a71aa1888695b076e2caf7f7148ebc8b4abcc0366209531e7da39f77aac46959"} Jul 07 14:15:59.281046 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:15:59.280988 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" event={"ID":"2a529a61-fb46-46dc-be67-869a613ea05f","Type":"ContainerStarted","Data":"9175bc2fb736dc59d4f0e4f36a6c7a1f29eec14d5f1f1a67b1cbdebb523609ee"} Jul 07 14:16:00.288121 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:00.288082 2560 generic.go:356] "Generic (PLEG): container finished" podID="9c235c71-4d6c-46ef-a8c5-252a2d6e1480" containerID="194731055c79428651ed0b87a4f5b2e60f3616609f1c203851f11a914b186302" exitCode=0 Jul 07 14:16:00.288523 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:00.288186 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" event={"ID":"9c235c71-4d6c-46ef-a8c5-252a2d6e1480","Type":"ContainerDied","Data":"194731055c79428651ed0b87a4f5b2e60f3616609f1c203851f11a914b186302"} Jul 07 14:16:00.290735 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:00.290709 2560 generic.go:356] "Generic (PLEG): container finished" podID="bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f" containerID="f59a25338aaa7681d8dc2f538b808e2c70cd6718610542db05b49c71244a1c4e" exitCode=0 Jul 07 14:16:00.290844 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:00.290808 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" event={"ID":"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f","Type":"ContainerDied","Data":"f59a25338aaa7681d8dc2f538b808e2c70cd6718610542db05b49c71244a1c4e"} Jul 07 14:16:00.292816 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:00.292790 2560 generic.go:356] "Generic (PLEG): container finished" podID="f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669" containerID="76189728220b157fa0dd38baab55f6304e58d4c10755400a165734b2cd118767" exitCode=0 Jul 07 14:16:00.292933 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:00.292830 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" event={"ID":"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669","Type":"ContainerDied","Data":"76189728220b157fa0dd38baab55f6304e58d4c10755400a165734b2cd118767"} Jul 07 14:16:01.300150 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.300115 2560 generic.go:356] "Generic (PLEG): container finished" podID="bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f" containerID="9a7e18dd8589c137675b41da73c2812c6f80399f3f2301d767e44ec20be4de86" exitCode=0 Jul 07 14:16:01.300573 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.300193 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" event={"ID":"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f","Type":"ContainerDied","Data":"9a7e18dd8589c137675b41da73c2812c6f80399f3f2301d767e44ec20be4de86"} Jul 07 14:16:01.302164 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.302142 2560 generic.go:356] "Generic (PLEG): container finished" podID="f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669" containerID="1372e4d2f69fd42130457cb362b045498c79f296a81c74d14bdcd50cd92c309a" exitCode=0 Jul 07 14:16:01.302304 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.302218 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" event={"ID":"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669","Type":"ContainerDied","Data":"1372e4d2f69fd42130457cb362b045498c79f296a81c74d14bdcd50cd92c309a"} Jul 07 14:16:01.304075 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.304057 2560 generic.go:356] "Generic (PLEG): container finished" podID="2a529a61-fb46-46dc-be67-869a613ea05f" containerID="0f4a2617301f0686741e0cf504849d7c581829205f9964aad528a61a9cdf4526" exitCode=0 Jul 07 14:16:01.304183 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.304136 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" event={"ID":"2a529a61-fb46-46dc-be67-869a613ea05f","Type":"ContainerDied","Data":"0f4a2617301f0686741e0cf504849d7c581829205f9964aad528a61a9cdf4526"} Jul 07 14:16:01.443722 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.443699 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" Jul 07 14:16:01.516202 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.516166 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-util\") pod \"9c235c71-4d6c-46ef-a8c5-252a2d6e1480\" (UID: \"9c235c71-4d6c-46ef-a8c5-252a2d6e1480\") " Jul 07 14:16:01.516382 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.516315 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-bundle\") pod \"9c235c71-4d6c-46ef-a8c5-252a2d6e1480\" (UID: \"9c235c71-4d6c-46ef-a8c5-252a2d6e1480\") " Jul 07 14:16:01.516382 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.516346 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4glq\" (UniqueName: \"kubernetes.io/projected/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-kube-api-access-w4glq\") pod \"9c235c71-4d6c-46ef-a8c5-252a2d6e1480\" (UID: \"9c235c71-4d6c-46ef-a8c5-252a2d6e1480\") " Jul 07 14:16:01.516926 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.516899 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-bundle" (OuterVolumeSpecName: "bundle") pod "9c235c71-4d6c-46ef-a8c5-252a2d6e1480" (UID: "9c235c71-4d6c-46ef-a8c5-252a2d6e1480"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:16:01.518679 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.518654 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-kube-api-access-w4glq" (OuterVolumeSpecName: "kube-api-access-w4glq") pod "9c235c71-4d6c-46ef-a8c5-252a2d6e1480" (UID: "9c235c71-4d6c-46ef-a8c5-252a2d6e1480"). InnerVolumeSpecName "kube-api-access-w4glq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:16:01.613737 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.613659 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-util" (OuterVolumeSpecName: "util") pod "9c235c71-4d6c-46ef-a8c5-252a2d6e1480" (UID: "9c235c71-4d6c-46ef-a8c5-252a2d6e1480"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:16:01.617100 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.617081 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-util\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:16:01.617164 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.617104 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-bundle\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:16:01.617164 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:01.617113 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4glq\" (UniqueName: \"kubernetes.io/projected/9c235c71-4d6c-46ef-a8c5-252a2d6e1480-kube-api-access-w4glq\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:16:02.310552 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.310521 2560 generic.go:356] "Generic (PLEG): container finished" podID="2a529a61-fb46-46dc-be67-869a613ea05f" containerID="d8738747f23dddd9711b6fcbae312490522f28a46c8327759a923ece4dd11390" exitCode=0 Jul 07 14:16:02.310942 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.310614 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" event={"ID":"2a529a61-fb46-46dc-be67-869a613ea05f","Type":"ContainerDied","Data":"d8738747f23dddd9711b6fcbae312490522f28a46c8327759a923ece4dd11390"} Jul 07 14:16:02.312923 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.312892 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" event={"ID":"9c235c71-4d6c-46ef-a8c5-252a2d6e1480","Type":"ContainerDied","Data":"f2ab1bc9e68aeb363a13dcbf7a117d07683c749c1064547883f9bd6a56cd23a6"} Jul 07 14:16:02.313146 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.313132 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2ab1bc9e68aeb363a13dcbf7a117d07683c749c1064547883f9bd6a56cd23a6" Jul 07 14:16:02.313382 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.313367 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9" Jul 07 14:16:02.455780 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.455756 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" Jul 07 14:16:02.479125 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.479101 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" Jul 07 14:16:02.522830 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.522800 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q2sr\" (UniqueName: \"kubernetes.io/projected/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-kube-api-access-7q2sr\") pod \"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f\" (UID: \"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f\") " Jul 07 14:16:02.522998 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.522855 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-bundle\") pod \"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f\" (UID: \"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f\") " Jul 07 14:16:02.522998 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.522895 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-bundle\") pod \"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669\" (UID: \"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669\") " Jul 07 14:16:02.522998 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.522963 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-util\") pod \"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669\" (UID: \"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669\") " Jul 07 14:16:02.522998 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.522990 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-util\") pod \"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f\" (UID: \"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f\") " Jul 07 14:16:02.523216 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.523031 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q7d7\" (UniqueName: \"kubernetes.io/projected/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-kube-api-access-5q7d7\") pod \"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669\" (UID: \"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669\") " Jul 07 14:16:02.523533 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.523506 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-bundle" (OuterVolumeSpecName: "bundle") pod "bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f" (UID: "bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:16:02.524003 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.523974 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-bundle" (OuterVolumeSpecName: "bundle") pod "f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669" (UID: "f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:16:02.525348 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.525314 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-kube-api-access-7q2sr" (OuterVolumeSpecName: "kube-api-access-7q2sr") pod "bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f" (UID: "bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f"). InnerVolumeSpecName "kube-api-access-7q2sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:16:02.525933 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.525910 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-kube-api-access-5q7d7" (OuterVolumeSpecName: "kube-api-access-5q7d7") pod "f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669" (UID: "f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669"). InnerVolumeSpecName "kube-api-access-5q7d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:16:02.529930 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.529901 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-util" (OuterVolumeSpecName: "util") pod "bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f" (UID: "bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:16:02.532970 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.532945 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-util" (OuterVolumeSpecName: "util") pod "f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669" (UID: "f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:16:02.624162 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.624068 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-util\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:16:02.624162 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.624101 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-util\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:16:02.624162 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.624112 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q7d7\" (UniqueName: \"kubernetes.io/projected/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-kube-api-access-5q7d7\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:16:02.624162 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.624124 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q2sr\" (UniqueName: \"kubernetes.io/projected/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-kube-api-access-7q2sr\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:16:02.624162 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.624133 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f-bundle\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:16:02.624162 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:02.624142 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669-bundle\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:16:03.320528 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.320497 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" Jul 07 14:16:03.320998 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.320497 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf" event={"ID":"bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f","Type":"ContainerDied","Data":"b06a34327e0fb7da8d1242829dac28b94dcdd3740762bcd49fb22a6a77ecb3d2"} Jul 07 14:16:03.320998 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.320612 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b06a34327e0fb7da8d1242829dac28b94dcdd3740762bcd49fb22a6a77ecb3d2" Jul 07 14:16:03.322403 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.322385 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" Jul 07 14:16:03.322520 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.322417 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2" event={"ID":"f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669","Type":"ContainerDied","Data":"41035837bce736d0f8089d63c0ccc02f444c1bf69f09e8de286c8f80daab5f8d"} Jul 07 14:16:03.322520 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.322453 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41035837bce736d0f8089d63c0ccc02f444c1bf69f09e8de286c8f80daab5f8d" Jul 07 14:16:03.467447 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.467423 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" Jul 07 14:16:03.531912 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.531877 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a529a61-fb46-46dc-be67-869a613ea05f-util\") pod \"2a529a61-fb46-46dc-be67-869a613ea05f\" (UID: \"2a529a61-fb46-46dc-be67-869a613ea05f\") " Jul 07 14:16:03.532103 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.531952 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmbjr\" (UniqueName: \"kubernetes.io/projected/2a529a61-fb46-46dc-be67-869a613ea05f-kube-api-access-hmbjr\") pod \"2a529a61-fb46-46dc-be67-869a613ea05f\" (UID: \"2a529a61-fb46-46dc-be67-869a613ea05f\") " Jul 07 14:16:03.532103 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.531993 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a529a61-fb46-46dc-be67-869a613ea05f-bundle\") pod \"2a529a61-fb46-46dc-be67-869a613ea05f\" (UID: \"2a529a61-fb46-46dc-be67-869a613ea05f\") " Jul 07 14:16:03.532538 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.532510 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a529a61-fb46-46dc-be67-869a613ea05f-bundle" (OuterVolumeSpecName: "bundle") pod "2a529a61-fb46-46dc-be67-869a613ea05f" (UID: "2a529a61-fb46-46dc-be67-869a613ea05f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:16:03.534302 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.534256 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a529a61-fb46-46dc-be67-869a613ea05f-kube-api-access-hmbjr" (OuterVolumeSpecName: "kube-api-access-hmbjr") pod "2a529a61-fb46-46dc-be67-869a613ea05f" (UID: "2a529a61-fb46-46dc-be67-869a613ea05f"). InnerVolumeSpecName "kube-api-access-hmbjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:16:03.538367 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.538344 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a529a61-fb46-46dc-be67-869a613ea05f-util" (OuterVolumeSpecName: "util") pod "2a529a61-fb46-46dc-be67-869a613ea05f" (UID: "2a529a61-fb46-46dc-be67-869a613ea05f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:16:03.633566 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.633483 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a529a61-fb46-46dc-be67-869a613ea05f-bundle\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:16:03.633566 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.633511 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a529a61-fb46-46dc-be67-869a613ea05f-util\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:16:03.633566 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:03.633519 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmbjr\" (UniqueName: \"kubernetes.io/projected/2a529a61-fb46-46dc-be67-869a613ea05f-kube-api-access-hmbjr\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:16:04.329223 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:04.329195 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" Jul 07 14:16:04.329223 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:04.329211 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj" event={"ID":"2a529a61-fb46-46dc-be67-869a613ea05f","Type":"ContainerDied","Data":"9175bc2fb736dc59d4f0e4f36a6c7a1f29eec14d5f1f1a67b1cbdebb523609ee"} Jul 07 14:16:04.329644 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:04.329254 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9175bc2fb736dc59d4f0e4f36a6c7a1f29eec14d5f1f1a67b1cbdebb523609ee" Jul 07 14:16:09.901587 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:09.901552 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:13.504015 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:13.503907 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:16.543895 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:16.543856 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:18.041451 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:18.041418 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:19.301621 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:19.301584 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:22.301089 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:22.301050 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:22.922743 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:22.922699 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:28.113072 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:28.113013 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:28.214183 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:28.214150 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:28.397029 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:28.396949 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:28.540861 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:28.540832 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:29.296772 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.296735 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b94988699-mrptp"] Jul 07 14:16:29.297144 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297082 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669" containerName="pull" Jul 07 14:16:29.297144 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297096 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669" containerName="pull" Jul 07 14:16:29.297144 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297108 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669" containerName="util" Jul 07 14:16:29.297144 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297114 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669" containerName="util" Jul 07 14:16:29.297144 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297121 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c235c71-4d6c-46ef-a8c5-252a2d6e1480" containerName="pull" Jul 07 14:16:29.297144 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297127 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c235c71-4d6c-46ef-a8c5-252a2d6e1480" containerName="pull" Jul 07 14:16:29.297144 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297135 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a529a61-fb46-46dc-be67-869a613ea05f" containerName="util" Jul 07 14:16:29.297144 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297140 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a529a61-fb46-46dc-be67-869a613ea05f" containerName="util" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297156 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a529a61-fb46-46dc-be67-869a613ea05f" containerName="pull" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297165 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a529a61-fb46-46dc-be67-869a613ea05f" containerName="pull" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297175 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a529a61-fb46-46dc-be67-869a613ea05f" containerName="extract" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297180 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a529a61-fb46-46dc-be67-869a613ea05f" containerName="extract" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297188 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f" containerName="util" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297194 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f" containerName="util" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297200 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f" containerName="extract" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297205 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f" containerName="extract" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297212 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c235c71-4d6c-46ef-a8c5-252a2d6e1480" containerName="extract" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297217 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c235c71-4d6c-46ef-a8c5-252a2d6e1480" containerName="extract" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297230 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669" containerName="extract" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297239 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669" containerName="extract" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297249 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f" containerName="pull" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297254 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f" containerName="pull" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297277 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c235c71-4d6c-46ef-a8c5-252a2d6e1480" containerName="util" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297283 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c235c71-4d6c-46ef-a8c5-252a2d6e1480" containerName="util" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297343 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a529a61-fb46-46dc-be67-869a613ea05f" containerName="extract" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297355 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669" containerName="extract" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297361 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f" containerName="extract" Jul 07 14:16:29.297413 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.297367 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c235c71-4d6c-46ef-a8c5-252a2d6e1480" containerName="extract" Jul 07 14:16:29.300600 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.300584 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.312759 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.312735 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b94988699-mrptp"] Jul 07 14:16:29.361357 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.361332 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:29.467038 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.467009 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a27c13ef-4536-428d-b28d-287cf1bddbc0-trusted-ca-bundle\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.467190 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.467049 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a27c13ef-4536-428d-b28d-287cf1bddbc0-oauth-serving-cert\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.467190 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.467079 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a27c13ef-4536-428d-b28d-287cf1bddbc0-console-config\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.467190 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.467117 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a27c13ef-4536-428d-b28d-287cf1bddbc0-console-serving-cert\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.467190 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.467158 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a27c13ef-4536-428d-b28d-287cf1bddbc0-console-oauth-config\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.467353 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.467214 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a27c13ef-4536-428d-b28d-287cf1bddbc0-service-ca\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.467353 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.467256 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld4mw\" (UniqueName: \"kubernetes.io/projected/a27c13ef-4536-428d-b28d-287cf1bddbc0-kube-api-access-ld4mw\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.568559 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.568487 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a27c13ef-4536-428d-b28d-287cf1bddbc0-console-oauth-config\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.568559 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.568523 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a27c13ef-4536-428d-b28d-287cf1bddbc0-service-ca\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.568559 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.568543 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld4mw\" (UniqueName: \"kubernetes.io/projected/a27c13ef-4536-428d-b28d-287cf1bddbc0-kube-api-access-ld4mw\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.568766 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.568565 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a27c13ef-4536-428d-b28d-287cf1bddbc0-trusted-ca-bundle\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.568766 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.568695 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a27c13ef-4536-428d-b28d-287cf1bddbc0-oauth-serving-cert\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.568766 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.568745 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a27c13ef-4536-428d-b28d-287cf1bddbc0-console-config\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.568917 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.568841 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a27c13ef-4536-428d-b28d-287cf1bddbc0-console-serving-cert\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.569312 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.569288 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a27c13ef-4536-428d-b28d-287cf1bddbc0-service-ca\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.569464 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.569443 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a27c13ef-4536-428d-b28d-287cf1bddbc0-trusted-ca-bundle\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.569464 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.569456 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a27c13ef-4536-428d-b28d-287cf1bddbc0-oauth-serving-cert\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.569645 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.569626 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a27c13ef-4536-428d-b28d-287cf1bddbc0-console-config\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.571154 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.571133 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a27c13ef-4536-428d-b28d-287cf1bddbc0-console-oauth-config\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.571240 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.571221 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a27c13ef-4536-428d-b28d-287cf1bddbc0-console-serving-cert\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.576383 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.576362 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld4mw\" (UniqueName: \"kubernetes.io/projected/a27c13ef-4536-428d-b28d-287cf1bddbc0-kube-api-access-ld4mw\") pod \"console-7b94988699-mrptp\" (UID: \"a27c13ef-4536-428d-b28d-287cf1bddbc0\") " pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.610410 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.610387 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:29.744507 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:29.744482 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b94988699-mrptp"] Jul 07 14:16:29.746944 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:16:29.746915 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27c13ef_4536_428d_b28d_287cf1bddbc0.slice/crio-18061f84f1f5160bb0986711d6ab7f1190236f40be0ae34bf835ffed563b6603 WatchSource:0}: Error finding container 18061f84f1f5160bb0986711d6ab7f1190236f40be0ae34bf835ffed563b6603: Status 404 returned error can't find the container with id 18061f84f1f5160bb0986711d6ab7f1190236f40be0ae34bf835ffed563b6603 Jul 07 14:16:30.469551 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:30.469515 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b94988699-mrptp" event={"ID":"a27c13ef-4536-428d-b28d-287cf1bddbc0","Type":"ContainerStarted","Data":"0449408387f98d6c7cccd9f2b92d70070833a3d3a5d6979b404b7949f721cf22"} Jul 07 14:16:30.469932 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:30.469572 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b94988699-mrptp" event={"ID":"a27c13ef-4536-428d-b28d-287cf1bddbc0","Type":"ContainerStarted","Data":"18061f84f1f5160bb0986711d6ab7f1190236f40be0ae34bf835ffed563b6603"} Jul 07 14:16:30.489922 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:30.489848 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b94988699-mrptp" podStartSLOduration=1.489827091 podStartE2EDuration="1.489827091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:16:30.489628026 +0000 UTC m=+548.921870548" watchObservedRunningTime="2026-07-07 14:16:30.489827091 +0000 UTC m=+548.922069593" Jul 07 14:16:31.906096 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:31.906060 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:32.009563 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:32.009529 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:36.073808 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:36.073778 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:36.201087 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:36.201057 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:39.090935 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.090901 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg"] Jul 07 14:16:39.096144 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.096124 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" Jul 07 14:16:39.098522 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.098503 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 14:16:39.098634 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.098573 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jgl9n\"" Jul 07 14:16:39.098634 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.098590 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 14:16:39.109371 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.109350 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg"] Jul 07 14:16:39.152830 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.152794 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66wms\" (UniqueName: \"kubernetes.io/projected/547f8ff1-66c5-4f09-a797-25e4188d652f-kube-api-access-66wms\") pod \"kuadrant-operator-controller-manager-86878c769f-gkkrg\" (UID: \"547f8ff1-66c5-4f09-a797-25e4188d652f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" Jul 07 14:16:39.152830 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.152830 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/547f8ff1-66c5-4f09-a797-25e4188d652f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gkkrg\" (UID: \"547f8ff1-66c5-4f09-a797-25e4188d652f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" Jul 07 14:16:39.254042 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.254009 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wms\" (UniqueName: \"kubernetes.io/projected/547f8ff1-66c5-4f09-a797-25e4188d652f-kube-api-access-66wms\") pod \"kuadrant-operator-controller-manager-86878c769f-gkkrg\" (UID: \"547f8ff1-66c5-4f09-a797-25e4188d652f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" Jul 07 14:16:39.254042 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.254043 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/547f8ff1-66c5-4f09-a797-25e4188d652f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gkkrg\" (UID: \"547f8ff1-66c5-4f09-a797-25e4188d652f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" Jul 07 14:16:39.254478 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.254456 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/547f8ff1-66c5-4f09-a797-25e4188d652f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gkkrg\" (UID: \"547f8ff1-66c5-4f09-a797-25e4188d652f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" Jul 07 14:16:39.266613 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.266593 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wms\" (UniqueName: \"kubernetes.io/projected/547f8ff1-66c5-4f09-a797-25e4188d652f-kube-api-access-66wms\") pod \"kuadrant-operator-controller-manager-86878c769f-gkkrg\" (UID: \"547f8ff1-66c5-4f09-a797-25e4188d652f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" Jul 07 14:16:39.407465 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.407394 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" Jul 07 14:16:39.549437 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.549410 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg"] Jul 07 14:16:39.551715 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:16:39.551685 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547f8ff1_66c5_4f09_a797_25e4188d652f.slice/crio-6cf4302a2c01480b665533e34d673346b04291f87cf3b899ce91b8247dc84ae5 WatchSource:0}: Error finding container 6cf4302a2c01480b665533e34d673346b04291f87cf3b899ce91b8247dc84ae5: Status 404 returned error can't find the container with id 6cf4302a2c01480b665533e34d673346b04291f87cf3b899ce91b8247dc84ae5 Jul 07 14:16:39.610905 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.610878 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:39.611020 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.610935 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:39.615541 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:39.615525 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:40.523990 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:40.523947 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" event={"ID":"547f8ff1-66c5-4f09-a797-25e4188d652f","Type":"ContainerStarted","Data":"6cf4302a2c01480b665533e34d673346b04291f87cf3b899ce91b8247dc84ae5"} Jul 07 14:16:40.528799 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:40.528772 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b94988699-mrptp" Jul 07 14:16:40.592252 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:40.591519 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67554b6fdf-zd82b"] Jul 07 14:16:40.699963 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:40.699920 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:40.802865 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:40.802832 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:41.015187 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:41.015157 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:42.739188 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:42.739156 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:42.845146 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:42.845115 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:44.604622 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:44.604571 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:44.704901 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:44.704871 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:44.761468 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:44.761437 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:45.555974 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:45.555938 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" event={"ID":"547f8ff1-66c5-4f09-a797-25e4188d652f","Type":"ContainerStarted","Data":"2b89196179757870c3ae8b6094588a74fdcf4994bbe19fc2b99f2e099429e3ae"} Jul 07 14:16:45.556157 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:45.556005 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" Jul 07 14:16:45.579279 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:45.579225 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" podStartSLOduration=1.004844736 podStartE2EDuration="6.579210208s" totalImagesPullingTime="5.574365472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:16:45.576803424 +0000 UTC m=+564.009045925" watchObservedRunningTime="2026-07-07 14:16:45.579210208 +0000 UTC m=+564.011452709" Jul 07 14:16:46.327073 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:46.327044 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:46.390821 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:46.390778 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:49.909122 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:49.909086 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:50.007685 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:50.007655 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:53.563325 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:53.563293 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:53.659092 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:53.659055 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:16:56.563223 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:16:56.563186 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" Jul 07 14:17:00.918105 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:00.918072 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:02.354574 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:02.354506 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:05.620673 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:05.620626 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-67554b6fdf-zd82b" podUID="c35722dd-f496-44ca-b846-46687b0f15dd" containerName="console" containerID="cri-o://273981c12ee6c4f9a036d67dc0a57db6dbfe6ff0b8577cd5486545e385de0b5c" gracePeriod=15 Jul 07 14:17:05.866909 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:05.866883 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:17:05.998887 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:05.998806 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-oauth-serving-cert\") pod \"c35722dd-f496-44ca-b846-46687b0f15dd\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " Jul 07 14:17:05.998887 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:05.998868 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vncz6\" (UniqueName: \"kubernetes.io/projected/c35722dd-f496-44ca-b846-46687b0f15dd-kube-api-access-vncz6\") pod \"c35722dd-f496-44ca-b846-46687b0f15dd\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " Jul 07 14:17:05.998887 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:05.998888 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-trusted-ca-bundle\") pod \"c35722dd-f496-44ca-b846-46687b0f15dd\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " Jul 07 14:17:05.999180 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:05.998913 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-console-config\") pod \"c35722dd-f496-44ca-b846-46687b0f15dd\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " Jul 07 14:17:05.999180 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:05.998940 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-service-ca\") pod \"c35722dd-f496-44ca-b846-46687b0f15dd\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " Jul 07 14:17:05.999180 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:05.999000 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c35722dd-f496-44ca-b846-46687b0f15dd-console-oauth-config\") pod \"c35722dd-f496-44ca-b846-46687b0f15dd\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " Jul 07 14:17:05.999180 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:05.999040 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c35722dd-f496-44ca-b846-46687b0f15dd-console-serving-cert\") pod \"c35722dd-f496-44ca-b846-46687b0f15dd\" (UID: \"c35722dd-f496-44ca-b846-46687b0f15dd\") " Jul 07 14:17:05.999412 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:05.999302 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c35722dd-f496-44ca-b846-46687b0f15dd" (UID: "c35722dd-f496-44ca-b846-46687b0f15dd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:17:05.999412 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:05.999350 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c35722dd-f496-44ca-b846-46687b0f15dd" (UID: "c35722dd-f496-44ca-b846-46687b0f15dd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:17:05.999412 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:05.999367 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-console-config" (OuterVolumeSpecName: "console-config") pod "c35722dd-f496-44ca-b846-46687b0f15dd" (UID: "c35722dd-f496-44ca-b846-46687b0f15dd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:17:05.999412 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:05.999375 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-service-ca" (OuterVolumeSpecName: "service-ca") pod "c35722dd-f496-44ca-b846-46687b0f15dd" (UID: "c35722dd-f496-44ca-b846-46687b0f15dd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:17:06.001156 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.001130 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35722dd-f496-44ca-b846-46687b0f15dd-kube-api-access-vncz6" (OuterVolumeSpecName: "kube-api-access-vncz6") pod "c35722dd-f496-44ca-b846-46687b0f15dd" (UID: "c35722dd-f496-44ca-b846-46687b0f15dd"). InnerVolumeSpecName "kube-api-access-vncz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:17:06.001377 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.001172 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c35722dd-f496-44ca-b846-46687b0f15dd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c35722dd-f496-44ca-b846-46687b0f15dd" (UID: "c35722dd-f496-44ca-b846-46687b0f15dd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:17:06.001377 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.001327 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c35722dd-f496-44ca-b846-46687b0f15dd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c35722dd-f496-44ca-b846-46687b0f15dd" (UID: "c35722dd-f496-44ca-b846-46687b0f15dd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:17:06.100473 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.100428 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c35722dd-f496-44ca-b846-46687b0f15dd-console-serving-cert\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:17:06.100473 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.100465 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-oauth-serving-cert\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:17:06.100473 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.100475 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vncz6\" (UniqueName: \"kubernetes.io/projected/c35722dd-f496-44ca-b846-46687b0f15dd-kube-api-access-vncz6\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:17:06.100473 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.100485 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-trusted-ca-bundle\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:17:06.100741 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.100495 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-console-config\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:17:06.100741 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.100504 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c35722dd-f496-44ca-b846-46687b0f15dd-service-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:17:06.100741 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.100512 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c35722dd-f496-44ca-b846-46687b0f15dd-console-oauth-config\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:17:06.142220 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.142187 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:06.242449 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.242419 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:06.673851 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.673816 2560 generic.go:356] "Generic (PLEG): container finished" podID="c35722dd-f496-44ca-b846-46687b0f15dd" containerID="273981c12ee6c4f9a036d67dc0a57db6dbfe6ff0b8577cd5486545e385de0b5c" exitCode=0 Jul 07 14:17:06.674313 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.673862 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67554b6fdf-zd82b" Jul 07 14:17:06.674313 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.673904 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67554b6fdf-zd82b" event={"ID":"c35722dd-f496-44ca-b846-46687b0f15dd","Type":"ContainerDied","Data":"273981c12ee6c4f9a036d67dc0a57db6dbfe6ff0b8577cd5486545e385de0b5c"} Jul 07 14:17:06.674313 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.673956 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67554b6fdf-zd82b" event={"ID":"c35722dd-f496-44ca-b846-46687b0f15dd","Type":"ContainerDied","Data":"ae9974052c1108bf559f75c1534bea8440a0da2d829f81356f587bba832fbfbb"} Jul 07 14:17:06.674313 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.673977 2560 scope.go:117] "RemoveContainer" containerID="273981c12ee6c4f9a036d67dc0a57db6dbfe6ff0b8577cd5486545e385de0b5c" Jul 07 14:17:06.683293 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.683247 2560 scope.go:117] "RemoveContainer" containerID="273981c12ee6c4f9a036d67dc0a57db6dbfe6ff0b8577cd5486545e385de0b5c" Jul 07 14:17:06.683591 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:17:06.683573 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273981c12ee6c4f9a036d67dc0a57db6dbfe6ff0b8577cd5486545e385de0b5c\": container with ID starting with 273981c12ee6c4f9a036d67dc0a57db6dbfe6ff0b8577cd5486545e385de0b5c not found: ID does not exist" containerID="273981c12ee6c4f9a036d67dc0a57db6dbfe6ff0b8577cd5486545e385de0b5c" Jul 07 14:17:06.683652 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.683601 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273981c12ee6c4f9a036d67dc0a57db6dbfe6ff0b8577cd5486545e385de0b5c"} err="failed to get container status \"273981c12ee6c4f9a036d67dc0a57db6dbfe6ff0b8577cd5486545e385de0b5c\": rpc error: code = NotFound desc = could not find container \"273981c12ee6c4f9a036d67dc0a57db6dbfe6ff0b8577cd5486545e385de0b5c\": container with ID starting with 273981c12ee6c4f9a036d67dc0a57db6dbfe6ff0b8577cd5486545e385de0b5c not found: ID does not exist" Jul 07 14:17:06.692380 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.692359 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67554b6fdf-zd82b"] Jul 07 14:17:06.696612 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.696592 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67554b6fdf-zd82b"] Jul 07 14:17:06.877755 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.877726 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:06.980252 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:06.980167 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:07.066018 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:07.065990 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:08.209104 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:08.209073 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c35722dd-f496-44ca-b846-46687b0f15dd" path="/var/lib/kubelet/pods/c35722dd-f496-44ca-b846-46687b0f15dd/volumes" Jul 07 14:17:10.549395 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:10.549364 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:10.650159 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:10.650130 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:13.102280 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:13.102233 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:13.202103 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:13.202071 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:14.371175 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:14.371143 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:14.470799 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:14.470768 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:15.472490 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:15.472458 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:15.576149 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:15.576121 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:15.679543 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:15.679515 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:16.331285 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:16.331229 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:16.401505 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:16.401479 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:21.956525 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:21.956495 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:22.055981 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:22.055956 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:26.353359 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:26.353325 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:26.454246 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:26.454208 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:26.563119 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:26.563089 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:26.655078 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:26.654996 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:31.724951 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:31.724915 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:37.229516 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:37.229486 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:50.106516 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:50.106434 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:50.562240 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:50.562209 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:57.291414 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:57.291377 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:57.414357 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:57.414319 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:57.780951 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:57.780863 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:57.877658 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:57.877624 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:57.958043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:57.958003 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:17:58.016855 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:17:58.016826 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:18:01.391539 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:01.391499 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:18:01.470680 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:01.470655 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:18:04.974062 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:04.974033 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:18:05.047922 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:05.047898 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:18:30.175228 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.175199 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:18:30.271578 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.271544 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:18:30.275748 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.275721 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-77476c4d87-vq4hq"] Jul 07 14:18:30.276051 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.276038 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c35722dd-f496-44ca-b846-46687b0f15dd" containerName="console" Jul 07 14:18:30.276098 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.276052 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c35722dd-f496-44ca-b846-46687b0f15dd" containerName="console" Jul 07 14:18:30.276139 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.276120 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c35722dd-f496-44ca-b846-46687b0f15dd" containerName="console" Jul 07 14:18:30.278492 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.278474 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-77476c4d87-vq4hq" Jul 07 14:18:30.280842 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.280824 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-kthmm\"" Jul 07 14:18:30.280962 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.280870 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 07 14:18:30.293699 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.293678 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-77476c4d87-vq4hq"] Jul 07 14:18:30.302652 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.302630 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f3a26c-a61e-49b0-9ea9-730ed0c4f483-openshift-service-ca\") pod \"maas-api-77476c4d87-vq4hq\" (UID: \"a8f3a26c-a61e-49b0-9ea9-730ed0c4f483\") " pod="opendatahub/maas-api-77476c4d87-vq4hq" Jul 07 14:18:30.302748 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.302710 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f3a26c-a61e-49b0-9ea9-730ed0c4f483-maas-api-tls\") pod \"maas-api-77476c4d87-vq4hq\" (UID: \"a8f3a26c-a61e-49b0-9ea9-730ed0c4f483\") " pod="opendatahub/maas-api-77476c4d87-vq4hq" Jul 07 14:18:30.302748 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.302741 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfqjv\" (UniqueName: \"kubernetes.io/projected/a8f3a26c-a61e-49b0-9ea9-730ed0c4f483-kube-api-access-xfqjv\") pod \"maas-api-77476c4d87-vq4hq\" (UID: \"a8f3a26c-a61e-49b0-9ea9-730ed0c4f483\") " pod="opendatahub/maas-api-77476c4d87-vq4hq" Jul 07 14:18:30.338845 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.338814 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:18:30.403712 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.403681 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f3a26c-a61e-49b0-9ea9-730ed0c4f483-maas-api-tls\") pod \"maas-api-77476c4d87-vq4hq\" (UID: \"a8f3a26c-a61e-49b0-9ea9-730ed0c4f483\") " pod="opendatahub/maas-api-77476c4d87-vq4hq" Jul 07 14:18:30.403836 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.403727 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfqjv\" (UniqueName: \"kubernetes.io/projected/a8f3a26c-a61e-49b0-9ea9-730ed0c4f483-kube-api-access-xfqjv\") pod \"maas-api-77476c4d87-vq4hq\" (UID: \"a8f3a26c-a61e-49b0-9ea9-730ed0c4f483\") " pod="opendatahub/maas-api-77476c4d87-vq4hq" Jul 07 14:18:30.403836 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.403760 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f3a26c-a61e-49b0-9ea9-730ed0c4f483-openshift-service-ca\") pod \"maas-api-77476c4d87-vq4hq\" (UID: \"a8f3a26c-a61e-49b0-9ea9-730ed0c4f483\") " pod="opendatahub/maas-api-77476c4d87-vq4hq" Jul 07 14:18:30.404397 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.404377 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f3a26c-a61e-49b0-9ea9-730ed0c4f483-openshift-service-ca\") pod \"maas-api-77476c4d87-vq4hq\" (UID: \"a8f3a26c-a61e-49b0-9ea9-730ed0c4f483\") " pod="opendatahub/maas-api-77476c4d87-vq4hq" Jul 07 14:18:30.406023 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.406002 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8f3a26c-a61e-49b0-9ea9-730ed0c4f483-maas-api-tls\") pod \"maas-api-77476c4d87-vq4hq\" (UID: \"a8f3a26c-a61e-49b0-9ea9-730ed0c4f483\") " pod="opendatahub/maas-api-77476c4d87-vq4hq" Jul 07 14:18:30.412024 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.412003 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfqjv\" (UniqueName: \"kubernetes.io/projected/a8f3a26c-a61e-49b0-9ea9-730ed0c4f483-kube-api-access-xfqjv\") pod \"maas-api-77476c4d87-vq4hq\" (UID: \"a8f3a26c-a61e-49b0-9ea9-730ed0c4f483\") " pod="opendatahub/maas-api-77476c4d87-vq4hq" Jul 07 14:18:30.588860 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.588833 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-77476c4d87-vq4hq" Jul 07 14:18:30.631632 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.631589 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:18:30.722286 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.722234 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:18:30.731691 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.731669 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-77476c4d87-vq4hq"] Jul 07 14:18:30.734544 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:18:30.734516 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f3a26c_a61e_49b0_9ea9_730ed0c4f483.slice/crio-3a9b29499c0310d05558b109ffc49d98b70f894c64547e546a2857bb1374a8f3 WatchSource:0}: Error finding container 3a9b29499c0310d05558b109ffc49d98b70f894c64547e546a2857bb1374a8f3: Status 404 returned error can't find the container with id 3a9b29499c0310d05558b109ffc49d98b70f894c64547e546a2857bb1374a8f3 Jul 07 14:18:30.736173 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:30.736142 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:18:31.092973 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:31.092935 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-77476c4d87-vq4hq" event={"ID":"a8f3a26c-a61e-49b0-9ea9-730ed0c4f483","Type":"ContainerStarted","Data":"3a9b29499c0310d05558b109ffc49d98b70f894c64547e546a2857bb1374a8f3"} Jul 07 14:18:33.111041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:33.111005 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-77476c4d87-vq4hq" event={"ID":"a8f3a26c-a61e-49b0-9ea9-730ed0c4f483","Type":"ContainerStarted","Data":"6068af60963ddc52933541de094b004683b8d6da7e96d83d28fbb4543b0d0463"} Jul 07 14:18:33.111477 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:33.111165 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-77476c4d87-vq4hq" Jul 07 14:18:33.129114 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:33.129072 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-77476c4d87-vq4hq" podStartSLOduration=1.308480194 podStartE2EDuration="3.129059091s" totalImagesPullingTime="1.820578897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:18:33.127078562 +0000 UTC m=+671.559321066" watchObservedRunningTime="2026-07-07 14:18:33.129059091 +0000 UTC m=+671.561301593" Jul 07 14:18:39.121504 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:39.121472 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-77476c4d87-vq4hq" Jul 07 14:18:40.091618 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:40.091586 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:18:40.183225 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:40.183193 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:18:58.428282 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.428231 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp"] Jul 07 14:18:58.431634 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.431619 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.434408 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.434387 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 14:18:58.434995 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.434978 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kcktd\"" Jul 07 14:18:58.435039 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.435023 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 07 14:18:58.435095 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.435026 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 14:18:58.441786 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.441765 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp"] Jul 07 14:18:58.450143 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.450124 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:18:58.538570 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.538549 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/773a4f6f-bfd0-41e3-830c-884bf82b28dd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.538713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.538577 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/773a4f6f-bfd0-41e3-830c-884bf82b28dd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.538713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.538607 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/773a4f6f-bfd0-41e3-830c-884bf82b28dd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.538713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.538639 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5hrk\" (UniqueName: \"kubernetes.io/projected/773a4f6f-bfd0-41e3-830c-884bf82b28dd-kube-api-access-d5hrk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.538713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.538658 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/773a4f6f-bfd0-41e3-830c-884bf82b28dd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.538884 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.538755 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/773a4f6f-bfd0-41e3-830c-884bf82b28dd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.639892 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.639863 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/773a4f6f-bfd0-41e3-830c-884bf82b28dd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.640023 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.639900 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5hrk\" (UniqueName: \"kubernetes.io/projected/773a4f6f-bfd0-41e3-830c-884bf82b28dd-kube-api-access-d5hrk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.640023 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.639920 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/773a4f6f-bfd0-41e3-830c-884bf82b28dd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.640023 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.639975 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/773a4f6f-bfd0-41e3-830c-884bf82b28dd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.640023 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.640020 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/773a4f6f-bfd0-41e3-830c-884bf82b28dd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.640234 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.640045 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/773a4f6f-bfd0-41e3-830c-884bf82b28dd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.640355 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.640248 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/773a4f6f-bfd0-41e3-830c-884bf82b28dd-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.640439 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.640418 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/773a4f6f-bfd0-41e3-830c-884bf82b28dd-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.640491 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.640475 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/773a4f6f-bfd0-41e3-830c-884bf82b28dd-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.642346 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.642327 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/773a4f6f-bfd0-41e3-830c-884bf82b28dd-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.642465 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.642450 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/773a4f6f-bfd0-41e3-830c-884bf82b28dd-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.647411 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.647390 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5hrk\" (UniqueName: \"kubernetes.io/projected/773a4f6f-bfd0-41e3-830c-884bf82b28dd-kube-api-access-d5hrk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp\" (UID: \"773a4f6f-bfd0-41e3-830c-884bf82b28dd\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.742987 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.742917 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:18:58.881765 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:58.881724 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp"] Jul 07 14:18:58.883987 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:18:58.883951 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773a4f6f_bfd0_41e3_830c_884bf82b28dd.slice/crio-61941d80c28992e0d5378bf11fc83fece359c4239a340ee15b6c35f080dd4b4a WatchSource:0}: Error finding container 61941d80c28992e0d5378bf11fc83fece359c4239a340ee15b6c35f080dd4b4a: Status 404 returned error can't find the container with id 61941d80c28992e0d5378bf11fc83fece359c4239a340ee15b6c35f080dd4b4a Jul 07 14:18:59.241133 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:18:59.241095 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" event={"ID":"773a4f6f-bfd0-41e3-830c-884bf82b28dd","Type":"ContainerStarted","Data":"61941d80c28992e0d5378bf11fc83fece359c4239a340ee15b6c35f080dd4b4a"} Jul 07 14:19:00.248863 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:00.248830 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" event={"ID":"773a4f6f-bfd0-41e3-830c-884bf82b28dd","Type":"ContainerStarted","Data":"e544642955617bc566928e901226ff80b3dbf55f374d631f7de59637faaa2078"} Jul 07 14:19:00.267101 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:00.267053 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" podStartSLOduration=1.023314022 podStartE2EDuration="2.267036969s" totalImagesPullingTime="1.243722947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:19:00.266815222 +0000 UTC m=+698.699057723" watchObservedRunningTime="2026-07-07 14:19:00.267036969 +0000 UTC m=+698.699279473" Jul 07 14:19:01.912060 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:01.912021 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:19:08.743225 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:08.743185 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:19:08.743623 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:08.743254 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:19:08.755558 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:08.755532 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:19:09.305670 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:09.305639 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp" Jul 07 14:19:16.132540 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:16.132441 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:19:28.711672 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:28.711639 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:19:37.456559 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:37.456492 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:19:51.516698 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.516664 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh"] Jul 07 14:19:51.519628 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.519609 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.521973 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.521954 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 07 14:19:51.529015 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.528995 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh"] Jul 07 14:19:51.614449 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.614413 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f59b563-8e29-40fa-ba00-4c9e86480a4a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.614449 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.614453 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f59b563-8e29-40fa-ba00-4c9e86480a4a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.614721 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.614477 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f59b563-8e29-40fa-ba00-4c9e86480a4a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.614721 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.614629 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f59b563-8e29-40fa-ba00-4c9e86480a4a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.614721 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.614671 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqkq\" (UniqueName: \"kubernetes.io/projected/8f59b563-8e29-40fa-ba00-4c9e86480a4a-kube-api-access-cvqkq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.614721 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.614696 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f59b563-8e29-40fa-ba00-4c9e86480a4a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.642772 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.642741 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:19:51.715981 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.715935 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f59b563-8e29-40fa-ba00-4c9e86480a4a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.715981 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.715975 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqkq\" (UniqueName: \"kubernetes.io/projected/8f59b563-8e29-40fa-ba00-4c9e86480a4a-kube-api-access-cvqkq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.716232 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.715999 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f59b563-8e29-40fa-ba00-4c9e86480a4a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.716232 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.716033 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f59b563-8e29-40fa-ba00-4c9e86480a4a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.716232 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.716057 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f59b563-8e29-40fa-ba00-4c9e86480a4a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.716232 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.716075 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f59b563-8e29-40fa-ba00-4c9e86480a4a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.716515 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.716490 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f59b563-8e29-40fa-ba00-4c9e86480a4a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.716577 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.716514 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f59b563-8e29-40fa-ba00-4c9e86480a4a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.716577 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.716545 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f59b563-8e29-40fa-ba00-4c9e86480a4a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.718489 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.718467 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f59b563-8e29-40fa-ba00-4c9e86480a4a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.718773 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.718753 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f59b563-8e29-40fa-ba00-4c9e86480a4a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.724355 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.724334 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqkq\" (UniqueName: \"kubernetes.io/projected/8f59b563-8e29-40fa-ba00-4c9e86480a4a-kube-api-access-cvqkq\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zqpfh\" (UID: \"8f59b563-8e29-40fa-ba00-4c9e86480a4a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.830435 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.830402 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:19:51.966905 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:51.966877 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh"] Jul 07 14:19:51.968532 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:19:51.968508 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f59b563_8e29_40fa_ba00_4c9e86480a4a.slice/crio-b3cb38dc0756fd407e1a454fdeb824d459c7cfd000218e7f2f02bd2a87bc137b WatchSource:0}: Error finding container b3cb38dc0756fd407e1a454fdeb824d459c7cfd000218e7f2f02bd2a87bc137b: Status 404 returned error can't find the container with id b3cb38dc0756fd407e1a454fdeb824d459c7cfd000218e7f2f02bd2a87bc137b Jul 07 14:19:52.512082 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:52.511997 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" event={"ID":"8f59b563-8e29-40fa-ba00-4c9e86480a4a","Type":"ContainerStarted","Data":"0093e6a0fffc97304f64184c144169094f834304b5c56f4c8311509db0fbc587"} Jul 07 14:19:52.512082 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:52.512044 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" event={"ID":"8f59b563-8e29-40fa-ba00-4c9e86480a4a","Type":"ContainerStarted","Data":"b3cb38dc0756fd407e1a454fdeb824d459c7cfd000218e7f2f02bd2a87bc137b"} Jul 07 14:19:52.529734 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:19:52.529683 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" podStartSLOduration=1.314582152 podStartE2EDuration="1.529668596s" totalImagesPullingTime="215.086444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:19:52.527070683 +0000 UTC m=+750.959313184" watchObservedRunningTime="2026-07-07 14:19:52.529668596 +0000 UTC m=+750.961911126" Jul 07 14:20:01.831189 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:20:01.831151 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:20:01.831601 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:20:01.831210 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:20:01.844462 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:20:01.844428 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:20:02.574343 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:20:02.574308 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zqpfh" Jul 07 14:20:03.036684 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:20:03.036604 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:20:47.908894 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:20:47.908804 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:20:56.657149 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:20:56.657114 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:20:57.360615 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:20:57.360584 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:21:05.019669 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:21:05.019624 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:21:05.123863 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:21:05.123836 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:21:06.105783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:21:06.105742 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:21:06.207388 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:21:06.207359 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:22:17.026880 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:22:17.026787 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:22:18.394536 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:22:18.394505 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:22:30.112537 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:22:30.112503 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:22:59.020817 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:22:59.020782 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:22:59.116808 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:22:59.116774 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:22:59.219910 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:22:59.219876 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:23:04.780848 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:23:04.780815 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:23:04.879451 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:23:04.879414 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:23:04.977785 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:23:04.977755 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:23:05.119956 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:23:05.119925 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:23:10.947367 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:23:10.947332 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:23:11.050679 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:23:11.050650 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:20.856049 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:25:20.856020 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:26:02.286302 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:26:02.286247 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:01.926394 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:01.926311 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg"] Jul 07 14:27:01.926943 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:01.926590 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" podUID="547f8ff1-66c5-4f09-a797-25e4188d652f" containerName="manager" containerID="cri-o://2b89196179757870c3ae8b6094588a74fdcf4994bbe19fc2b99f2e099429e3ae" gracePeriod=10 Jul 07 14:27:04.179673 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.179644 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" Jul 07 14:27:04.224713 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.224679 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66wms\" (UniqueName: \"kubernetes.io/projected/547f8ff1-66c5-4f09-a797-25e4188d652f-kube-api-access-66wms\") pod \"547f8ff1-66c5-4f09-a797-25e4188d652f\" (UID: \"547f8ff1-66c5-4f09-a797-25e4188d652f\") " Jul 07 14:27:04.224891 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.224791 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/547f8ff1-66c5-4f09-a797-25e4188d652f-extensions-socket-volume\") pod \"547f8ff1-66c5-4f09-a797-25e4188d652f\" (UID: \"547f8ff1-66c5-4f09-a797-25e4188d652f\") " Jul 07 14:27:04.225171 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.225147 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547f8ff1-66c5-4f09-a797-25e4188d652f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "547f8ff1-66c5-4f09-a797-25e4188d652f" (UID: "547f8ff1-66c5-4f09-a797-25e4188d652f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:27:04.226935 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.226899 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547f8ff1-66c5-4f09-a797-25e4188d652f-kube-api-access-66wms" (OuterVolumeSpecName: "kube-api-access-66wms") pod "547f8ff1-66c5-4f09-a797-25e4188d652f" (UID: "547f8ff1-66c5-4f09-a797-25e4188d652f"). InnerVolumeSpecName "kube-api-access-66wms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:27:04.325664 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.325631 2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/547f8ff1-66c5-4f09-a797-25e4188d652f-extensions-socket-volume\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:27:04.325664 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.325659 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66wms\" (UniqueName: \"kubernetes.io/projected/547f8ff1-66c5-4f09-a797-25e4188d652f-kube-api-access-66wms\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:27:04.750685 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.750595 2560 generic.go:356] "Generic (PLEG): container finished" podID="547f8ff1-66c5-4f09-a797-25e4188d652f" containerID="2b89196179757870c3ae8b6094588a74fdcf4994bbe19fc2b99f2e099429e3ae" exitCode=0 Jul 07 14:27:04.750685 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.750639 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" Jul 07 14:27:04.750685 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.750657 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" event={"ID":"547f8ff1-66c5-4f09-a797-25e4188d652f","Type":"ContainerDied","Data":"2b89196179757870c3ae8b6094588a74fdcf4994bbe19fc2b99f2e099429e3ae"} Jul 07 14:27:04.750893 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.750700 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg" event={"ID":"547f8ff1-66c5-4f09-a797-25e4188d652f","Type":"ContainerDied","Data":"6cf4302a2c01480b665533e34d673346b04291f87cf3b899ce91b8247dc84ae5"} Jul 07 14:27:04.750893 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.750720 2560 scope.go:117] "RemoveContainer" containerID="2b89196179757870c3ae8b6094588a74fdcf4994bbe19fc2b99f2e099429e3ae" Jul 07 14:27:04.761050 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.761009 2560 scope.go:117] "RemoveContainer" containerID="2b89196179757870c3ae8b6094588a74fdcf4994bbe19fc2b99f2e099429e3ae" Jul 07 14:27:04.761350 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:27:04.761317 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b89196179757870c3ae8b6094588a74fdcf4994bbe19fc2b99f2e099429e3ae\": container with ID starting with 2b89196179757870c3ae8b6094588a74fdcf4994bbe19fc2b99f2e099429e3ae not found: ID does not exist" containerID="2b89196179757870c3ae8b6094588a74fdcf4994bbe19fc2b99f2e099429e3ae" Jul 07 14:27:04.761454 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.761353 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b89196179757870c3ae8b6094588a74fdcf4994bbe19fc2b99f2e099429e3ae"} err="failed to get container status \"2b89196179757870c3ae8b6094588a74fdcf4994bbe19fc2b99f2e099429e3ae\": rpc error: code = NotFound desc = could not find container \"2b89196179757870c3ae8b6094588a74fdcf4994bbe19fc2b99f2e099429e3ae\": container with ID starting with 2b89196179757870c3ae8b6094588a74fdcf4994bbe19fc2b99f2e099429e3ae not found: ID does not exist" Jul 07 14:27:04.772150 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.772120 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg"] Jul 07 14:27:04.775257 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:04.775234 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gkkrg"] Jul 07 14:27:06.209120 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:06.209086 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="547f8ff1-66c5-4f09-a797-25e4188d652f" path="/var/lib/kubelet/pods/547f8ff1-66c5-4f09-a797-25e4188d652f/volumes" Jul 07 14:27:30.447371 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:30.447336 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:32.195666 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:27:32.195631 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:28:03.096040 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.096006 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs"] Jul 07 14:28:03.096538 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.096371 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="547f8ff1-66c5-4f09-a797-25e4188d652f" containerName="manager" Jul 07 14:28:03.096538 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.096384 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="547f8ff1-66c5-4f09-a797-25e4188d652f" containerName="manager" Jul 07 14:28:03.096538 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.096446 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="547f8ff1-66c5-4f09-a797-25e4188d652f" containerName="manager" Jul 07 14:28:03.108462 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.108437 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs" Jul 07 14:28:03.108583 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.108509 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs"] Jul 07 14:28:03.110897 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.110874 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jgl9n\"" Jul 07 14:28:03.111011 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.110880 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 14:28:03.111011 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.110935 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 14:28:03.255307 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.255246 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wrsbs\" (UID: \"7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs" Jul 07 14:28:03.255479 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.255317 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjbj\" (UniqueName: \"kubernetes.io/projected/7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8-kube-api-access-6xjbj\") pod \"kuadrant-operator-controller-manager-86878c769f-wrsbs\" (UID: \"7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs" Jul 07 14:28:03.356208 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.356137 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wrsbs\" (UID: \"7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs" Jul 07 14:28:03.356208 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.356199 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjbj\" (UniqueName: \"kubernetes.io/projected/7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8-kube-api-access-6xjbj\") pod \"kuadrant-operator-controller-manager-86878c769f-wrsbs\" (UID: \"7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs" Jul 07 14:28:03.356582 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.356551 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wrsbs\" (UID: \"7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs" Jul 07 14:28:03.366123 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.366107 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjbj\" (UniqueName: \"kubernetes.io/projected/7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8-kube-api-access-6xjbj\") pod \"kuadrant-operator-controller-manager-86878c769f-wrsbs\" (UID: \"7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs" Jul 07 14:28:03.418628 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.418608 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs" Jul 07 14:28:03.554329 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:03.554306 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs"] Jul 07 14:28:03.556240 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:28:03.556217 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7601e6d6_c27c_4522_a0fa_6d3fb8d7c9f8.slice/crio-655b3d309d97f93dd780fbe9851088592ee7a0a6dcde8259466afd38a1825623 WatchSource:0}: Error finding container 655b3d309d97f93dd780fbe9851088592ee7a0a6dcde8259466afd38a1825623: Status 404 returned error can't find the container with id 655b3d309d97f93dd780fbe9851088592ee7a0a6dcde8259466afd38a1825623 Jul 07 14:28:04.046716 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:04.046681 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs" event={"ID":"7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8","Type":"ContainerStarted","Data":"37d37ce3f830c7af773e5312d4e76a181fc6228d99e48ceed5a28988f621bd72"} Jul 07 14:28:04.046885 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:04.046744 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs" Jul 07 14:28:04.046885 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:04.046763 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs" event={"ID":"7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8","Type":"ContainerStarted","Data":"655b3d309d97f93dd780fbe9851088592ee7a0a6dcde8259466afd38a1825623"} Jul 07 14:28:04.067076 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:04.067023 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs" podStartSLOduration=1.067004756 podStartE2EDuration="1.067004756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:28:04.064430597 +0000 UTC m=+1242.496673098" watchObservedRunningTime="2026-07-07 14:28:04.067004756 +0000 UTC m=+1242.499247258" Jul 07 14:28:15.054097 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:28:15.054058 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wrsbs" Jul 07 14:30:08.376828 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:30:08.376745 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:12.437647 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:32:12.437603 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:31.046763 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:32:31.046730 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:31.100197 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:32:31.100132 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:45.721284 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:45.721224 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:47.309039 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:47.309001 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:51.830942 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:51.830913 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:51.887596 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:51.887539 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:52.027127 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:52.027077 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:52.125698 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:52.125615 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:53.205649 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.205613 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:53.300851 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.300816 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:53.358730 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.358698 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x"] Jul 07 14:33:53.361323 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.361301 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" Jul 07 14:33:53.363883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.363862 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-9d688a01-serving-cert\"" Jul 07 14:33:53.372725 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.372703 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x"] Jul 07 14:33:53.391956 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.391920 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gclgb\" (UniqueName: \"kubernetes.io/projected/493ffdd1-7746-4262-b7c1-14382d275e07-kube-api-access-gclgb\") pod \"maas-api-e2e-ait-9d688a01-7c9576b987-8w74x\" (UID: \"493ffdd1-7746-4262-b7c1-14382d275e07\") " pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" Jul 07 14:33:53.391956 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.391958 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/493ffdd1-7746-4262-b7c1-14382d275e07-maas-api-tls\") pod \"maas-api-e2e-ait-9d688a01-7c9576b987-8w74x\" (UID: \"493ffdd1-7746-4262-b7c1-14382d275e07\") " pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" Jul 07 14:33:53.392180 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.391976 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/493ffdd1-7746-4262-b7c1-14382d275e07-openshift-service-ca\") pod \"maas-api-e2e-ait-9d688a01-7c9576b987-8w74x\" (UID: \"493ffdd1-7746-4262-b7c1-14382d275e07\") " pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" Jul 07 14:33:53.400399 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.400375 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:53.493001 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.492906 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gclgb\" (UniqueName: \"kubernetes.io/projected/493ffdd1-7746-4262-b7c1-14382d275e07-kube-api-access-gclgb\") pod \"maas-api-e2e-ait-9d688a01-7c9576b987-8w74x\" (UID: \"493ffdd1-7746-4262-b7c1-14382d275e07\") " pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" Jul 07 14:33:53.493001 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.492949 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/493ffdd1-7746-4262-b7c1-14382d275e07-maas-api-tls\") pod \"maas-api-e2e-ait-9d688a01-7c9576b987-8w74x\" (UID: \"493ffdd1-7746-4262-b7c1-14382d275e07\") " pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" Jul 07 14:33:53.493001 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.492967 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/493ffdd1-7746-4262-b7c1-14382d275e07-openshift-service-ca\") pod \"maas-api-e2e-ait-9d688a01-7c9576b987-8w74x\" (UID: \"493ffdd1-7746-4262-b7c1-14382d275e07\") " pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" Jul 07 14:33:53.493327 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:33:53.493073 2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-9d688a01-serving-cert: secret "maas-api-e2e-ait-9d688a01-serving-cert" not found Jul 07 14:33:53.493327 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:33:53.493154 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/493ffdd1-7746-4262-b7c1-14382d275e07-maas-api-tls podName:493ffdd1-7746-4262-b7c1-14382d275e07 nodeName:}" failed. No retries permitted until 2026-07-07 14:33:53.993137989 +0000 UTC m=+1592.425380472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/493ffdd1-7746-4262-b7c1-14382d275e07-maas-api-tls") pod "maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" (UID: "493ffdd1-7746-4262-b7c1-14382d275e07") : secret "maas-api-e2e-ait-9d688a01-serving-cert" not found Jul 07 14:33:53.493616 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.493599 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/493ffdd1-7746-4262-b7c1-14382d275e07-openshift-service-ca\") pod \"maas-api-e2e-ait-9d688a01-7c9576b987-8w74x\" (UID: \"493ffdd1-7746-4262-b7c1-14382d275e07\") " pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" Jul 07 14:33:53.501116 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.501088 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gclgb\" (UniqueName: \"kubernetes.io/projected/493ffdd1-7746-4262-b7c1-14382d275e07-kube-api-access-gclgb\") pod \"maas-api-e2e-ait-9d688a01-7c9576b987-8w74x\" (UID: \"493ffdd1-7746-4262-b7c1-14382d275e07\") " pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" Jul 07 14:33:53.656170 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.656135 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:53.849503 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.849467 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:53.931394 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.931363 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:53.998059 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:53.998029 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/493ffdd1-7746-4262-b7c1-14382d275e07-maas-api-tls\") pod \"maas-api-e2e-ait-9d688a01-7c9576b987-8w74x\" (UID: \"493ffdd1-7746-4262-b7c1-14382d275e07\") " pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" Jul 07 14:33:54.000573 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:54.000549 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/493ffdd1-7746-4262-b7c1-14382d275e07-maas-api-tls\") pod \"maas-api-e2e-ait-9d688a01-7c9576b987-8w74x\" (UID: \"493ffdd1-7746-4262-b7c1-14382d275e07\") " pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" Jul 07 14:33:54.272864 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:54.272761 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" Jul 07 14:33:54.412033 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:54.412011 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x"] Jul 07 14:33:54.414258 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:33:54.414235 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493ffdd1_7746_4262_b7c1_14382d275e07.slice/crio-176c7280dba457f8faa9e09f73eb8036a4386c1959798532d7bf28a39955dbba WatchSource:0}: Error finding container 176c7280dba457f8faa9e09f73eb8036a4386c1959798532d7bf28a39955dbba: Status 404 returned error can't find the container with id 176c7280dba457f8faa9e09f73eb8036a4386c1959798532d7bf28a39955dbba Jul 07 14:33:54.415703 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:54.415688 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:33:54.877778 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:54.877741 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" event={"ID":"493ffdd1-7746-4262-b7c1-14382d275e07","Type":"ContainerStarted","Data":"176c7280dba457f8faa9e09f73eb8036a4386c1959798532d7bf28a39955dbba"} Jul 07 14:33:55.552974 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:55.552941 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:55.884618 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:55.884577 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" event={"ID":"493ffdd1-7746-4262-b7c1-14382d275e07","Type":"ContainerStarted","Data":"36352ca56d10cb1c6320520c61b494bfa5571835eb3fb37b3df04903b033e43e"} Jul 07 14:33:55.884802 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:55.884653 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" Jul 07 14:33:55.901973 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:55.901896 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" podStartSLOduration=2.332215836 podStartE2EDuration="2.901880092s" totalImagesPullingTime="569.664256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:33:55.899341251 +0000 UTC m=+1594.331583770" watchObservedRunningTime="2026-07-07 14:33:55.901880092 +0000 UTC m=+1594.334122594" Jul 07 14:33:58.202531 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.202501 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:58.389437 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.389397 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:58.495853 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.495777 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:58.496740 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.496715 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x"] Jul 07 14:33:58.496973 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.496924 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" podUID="493ffdd1-7746-4262-b7c1-14382d275e07" containerName="maas-api" containerID="cri-o://36352ca56d10cb1c6320520c61b494bfa5571835eb3fb37b3df04903b033e43e" gracePeriod=30 Jul 07 14:33:58.563723 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.563696 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:58.740759 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.740736 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" Jul 07 14:33:58.845407 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.845381 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/493ffdd1-7746-4262-b7c1-14382d275e07-maas-api-tls\") pod \"493ffdd1-7746-4262-b7c1-14382d275e07\" (UID: \"493ffdd1-7746-4262-b7c1-14382d275e07\") " Jul 07 14:33:58.845573 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.845473 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/493ffdd1-7746-4262-b7c1-14382d275e07-openshift-service-ca\") pod \"493ffdd1-7746-4262-b7c1-14382d275e07\" (UID: \"493ffdd1-7746-4262-b7c1-14382d275e07\") " Jul 07 14:33:58.845573 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.845503 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gclgb\" (UniqueName: \"kubernetes.io/projected/493ffdd1-7746-4262-b7c1-14382d275e07-kube-api-access-gclgb\") pod \"493ffdd1-7746-4262-b7c1-14382d275e07\" (UID: \"493ffdd1-7746-4262-b7c1-14382d275e07\") " Jul 07 14:33:58.845883 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.845856 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493ffdd1-7746-4262-b7c1-14382d275e07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "493ffdd1-7746-4262-b7c1-14382d275e07" (UID: "493ffdd1-7746-4262-b7c1-14382d275e07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:33:58.847768 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.847737 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493ffdd1-7746-4262-b7c1-14382d275e07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "493ffdd1-7746-4262-b7c1-14382d275e07" (UID: "493ffdd1-7746-4262-b7c1-14382d275e07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:33:58.847863 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.847801 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493ffdd1-7746-4262-b7c1-14382d275e07-kube-api-access-gclgb" (OuterVolumeSpecName: "kube-api-access-gclgb") pod "493ffdd1-7746-4262-b7c1-14382d275e07" (UID: "493ffdd1-7746-4262-b7c1-14382d275e07"). InnerVolumeSpecName "kube-api-access-gclgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:33:58.906525 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.906485 2560 generic.go:356] "Generic (PLEG): container finished" podID="493ffdd1-7746-4262-b7c1-14382d275e07" containerID="36352ca56d10cb1c6320520c61b494bfa5571835eb3fb37b3df04903b033e43e" exitCode=0 Jul 07 14:33:58.906715 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.906537 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" Jul 07 14:33:58.906715 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.906576 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" event={"ID":"493ffdd1-7746-4262-b7c1-14382d275e07","Type":"ContainerDied","Data":"36352ca56d10cb1c6320520c61b494bfa5571835eb3fb37b3df04903b033e43e"} Jul 07 14:33:58.906715 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.906631 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x" event={"ID":"493ffdd1-7746-4262-b7c1-14382d275e07","Type":"ContainerDied","Data":"176c7280dba457f8faa9e09f73eb8036a4386c1959798532d7bf28a39955dbba"} Jul 07 14:33:58.906715 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.906659 2560 scope.go:117] "RemoveContainer" containerID="36352ca56d10cb1c6320520c61b494bfa5571835eb3fb37b3df04903b033e43e" Jul 07 14:33:58.916816 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.916801 2560 scope.go:117] "RemoveContainer" containerID="36352ca56d10cb1c6320520c61b494bfa5571835eb3fb37b3df04903b033e43e" Jul 07 14:33:58.917072 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:33:58.917055 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36352ca56d10cb1c6320520c61b494bfa5571835eb3fb37b3df04903b033e43e\": container with ID starting with 36352ca56d10cb1c6320520c61b494bfa5571835eb3fb37b3df04903b033e43e not found: ID does not exist" containerID="36352ca56d10cb1c6320520c61b494bfa5571835eb3fb37b3df04903b033e43e" Jul 07 14:33:58.917137 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.917083 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36352ca56d10cb1c6320520c61b494bfa5571835eb3fb37b3df04903b033e43e"} err="failed to get container status \"36352ca56d10cb1c6320520c61b494bfa5571835eb3fb37b3df04903b033e43e\": rpc error: code = NotFound desc = could not find container \"36352ca56d10cb1c6320520c61b494bfa5571835eb3fb37b3df04903b033e43e\": container with ID starting with 36352ca56d10cb1c6320520c61b494bfa5571835eb3fb37b3df04903b033e43e not found: ID does not exist" Jul 07 14:33:58.928553 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.928531 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x"] Jul 07 14:33:58.933436 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.933413 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-9d688a01-7c9576b987-8w74x"] Jul 07 14:33:58.946243 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.946225 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/493ffdd1-7746-4262-b7c1-14382d275e07-openshift-service-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:33:58.946341 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.946246 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gclgb\" (UniqueName: \"kubernetes.io/projected/493ffdd1-7746-4262-b7c1-14382d275e07-kube-api-access-gclgb\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:33:58.946341 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:33:58.946257 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/493ffdd1-7746-4262-b7c1-14382d275e07-maas-api-tls\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:34:00.208642 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:00.208609 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="493ffdd1-7746-4262-b7c1-14382d275e07" path="/var/lib/kubelet/pods/493ffdd1-7746-4262-b7c1-14382d275e07/volumes" Jul 07 14:34:01.888675 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:01.888642 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:04.392650 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:04.392614 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:04.464657 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:04.464620 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:04.606315 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:04.606288 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:04.706003 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:04.705924 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:05.694482 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.694452 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:05.755907 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.755878 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2"] Jul 07 14:34:05.756248 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.756235 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493ffdd1-7746-4262-b7c1-14382d275e07" containerName="maas-api" Jul 07 14:34:05.756355 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.756249 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="493ffdd1-7746-4262-b7c1-14382d275e07" containerName="maas-api" Jul 07 14:34:05.756355 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.756338 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="493ffdd1-7746-4262-b7c1-14382d275e07" containerName="maas-api" Jul 07 14:34:05.763247 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.763225 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" Jul 07 14:34:05.765954 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.765924 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-42700fcd-serving-cert\"" Jul 07 14:34:05.769481 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.769457 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2"] Jul 07 14:34:05.806443 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.806411 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:05.811510 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.811491 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08daeede-e190-4e11-a04c-8f6d50f2bca4-maas-api-tls\") pod \"maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2\" (UID: \"08daeede-e190-4e11-a04c-8f6d50f2bca4\") " pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" Jul 07 14:34:05.811594 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.811521 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5lgv\" (UniqueName: \"kubernetes.io/projected/08daeede-e190-4e11-a04c-8f6d50f2bca4-kube-api-access-z5lgv\") pod \"maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2\" (UID: \"08daeede-e190-4e11-a04c-8f6d50f2bca4\") " pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" Jul 07 14:34:05.811594 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.811579 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08daeede-e190-4e11-a04c-8f6d50f2bca4-openshift-service-ca\") pod \"maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2\" (UID: \"08daeede-e190-4e11-a04c-8f6d50f2bca4\") " pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" Jul 07 14:34:05.907783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.907754 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:05.912962 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.912938 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08daeede-e190-4e11-a04c-8f6d50f2bca4-maas-api-tls\") pod \"maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2\" (UID: \"08daeede-e190-4e11-a04c-8f6d50f2bca4\") " pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" Jul 07 14:34:05.913095 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.912969 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5lgv\" (UniqueName: \"kubernetes.io/projected/08daeede-e190-4e11-a04c-8f6d50f2bca4-kube-api-access-z5lgv\") pod \"maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2\" (UID: \"08daeede-e190-4e11-a04c-8f6d50f2bca4\") " pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" Jul 07 14:34:05.913095 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.913046 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08daeede-e190-4e11-a04c-8f6d50f2bca4-openshift-service-ca\") pod \"maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2\" (UID: \"08daeede-e190-4e11-a04c-8f6d50f2bca4\") " pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" Jul 07 14:34:05.913095 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:34:05.913082 2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-42700fcd-serving-cert: secret "maas-api-e2e-ait-42700fcd-serving-cert" not found Jul 07 14:34:05.913223 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:34:05.913143 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08daeede-e190-4e11-a04c-8f6d50f2bca4-maas-api-tls podName:08daeede-e190-4e11-a04c-8f6d50f2bca4 nodeName:}" failed. No retries permitted until 2026-07-07 14:34:06.413122967 +0000 UTC m=+1604.845365450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08daeede-e190-4e11-a04c-8f6d50f2bca4-maas-api-tls") pod "maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" (UID: "08daeede-e190-4e11-a04c-8f6d50f2bca4") : secret "maas-api-e2e-ait-42700fcd-serving-cert" not found Jul 07 14:34:05.913740 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.913722 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08daeede-e190-4e11-a04c-8f6d50f2bca4-openshift-service-ca\") pod \"maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2\" (UID: \"08daeede-e190-4e11-a04c-8f6d50f2bca4\") " pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" Jul 07 14:34:05.922496 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.922478 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5lgv\" (UniqueName: \"kubernetes.io/projected/08daeede-e190-4e11-a04c-8f6d50f2bca4-kube-api-access-z5lgv\") pod \"maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2\" (UID: \"08daeede-e190-4e11-a04c-8f6d50f2bca4\") " pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" Jul 07 14:34:05.995290 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:05.995197 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:06.322861 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:06.322799 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:06.417505 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:06.417472 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08daeede-e190-4e11-a04c-8f6d50f2bca4-maas-api-tls\") pod \"maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2\" (UID: \"08daeede-e190-4e11-a04c-8f6d50f2bca4\") " pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" Jul 07 14:34:06.420318 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:06.420245 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08daeede-e190-4e11-a04c-8f6d50f2bca4-maas-api-tls\") pod \"maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2\" (UID: \"08daeede-e190-4e11-a04c-8f6d50f2bca4\") " pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" Jul 07 14:34:06.423440 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:06.423418 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:06.676648 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:06.676558 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" Jul 07 14:34:06.818196 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:06.818167 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2"] Jul 07 14:34:06.820350 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:34:06.820324 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08daeede_e190_4e11_a04c_8f6d50f2bca4.slice/crio-14dcad91d1d4801c5106f0f23a84fe33a2aa086e90a84adce695b603dd97faec WatchSource:0}: Error finding container 14dcad91d1d4801c5106f0f23a84fe33a2aa086e90a84adce695b603dd97faec: Status 404 returned error can't find the container with id 14dcad91d1d4801c5106f0f23a84fe33a2aa086e90a84adce695b603dd97faec Jul 07 14:34:06.955621 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:06.955533 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" event={"ID":"08daeede-e190-4e11-a04c-8f6d50f2bca4","Type":"ContainerStarted","Data":"14dcad91d1d4801c5106f0f23a84fe33a2aa086e90a84adce695b603dd97faec"} Jul 07 14:34:07.735476 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:07.735430 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:07.962804 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:07.962774 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" event={"ID":"08daeede-e190-4e11-a04c-8f6d50f2bca4","Type":"ContainerStarted","Data":"01a0bb1b180c0facb1fa69af3cb202959b7d3fbab4fc8b16cdb8c9a27aa5db8e"} Jul 07 14:34:07.968968 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:07.963305 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" Jul 07 14:34:07.982799 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:07.982697 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" podStartSLOduration=2.541003634 podStartE2EDuration="2.982682718s" totalImagesPullingTime="441.679084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:07.981614376 +0000 UTC m=+1606.413856878" watchObservedRunningTime="2026-07-07 14:34:07.982682718 +0000 UTC m=+1606.414925219" Jul 07 14:34:10.813836 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:10.813804 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:11.156033 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.155896 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2"] Jul 07 14:34:11.156239 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.156157 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" podUID="08daeede-e190-4e11-a04c-8f6d50f2bca4" containerName="maas-api" containerID="cri-o://01a0bb1b180c0facb1fa69af3cb202959b7d3fbab4fc8b16cdb8c9a27aa5db8e" gracePeriod=30 Jul 07 14:34:11.180716 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.180689 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:11.408746 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.408682 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" Jul 07 14:34:11.467850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.467813 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5lgv\" (UniqueName: \"kubernetes.io/projected/08daeede-e190-4e11-a04c-8f6d50f2bca4-kube-api-access-z5lgv\") pod \"08daeede-e190-4e11-a04c-8f6d50f2bca4\" (UID: \"08daeede-e190-4e11-a04c-8f6d50f2bca4\") " Jul 07 14:34:11.467994 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.467915 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08daeede-e190-4e11-a04c-8f6d50f2bca4-openshift-service-ca\") pod \"08daeede-e190-4e11-a04c-8f6d50f2bca4\" (UID: \"08daeede-e190-4e11-a04c-8f6d50f2bca4\") " Jul 07 14:34:11.467994 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.467941 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08daeede-e190-4e11-a04c-8f6d50f2bca4-maas-api-tls\") pod \"08daeede-e190-4e11-a04c-8f6d50f2bca4\" (UID: \"08daeede-e190-4e11-a04c-8f6d50f2bca4\") " Jul 07 14:34:11.468323 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.468247 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08daeede-e190-4e11-a04c-8f6d50f2bca4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "08daeede-e190-4e11-a04c-8f6d50f2bca4" (UID: "08daeede-e190-4e11-a04c-8f6d50f2bca4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:34:11.469943 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.469919 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08daeede-e190-4e11-a04c-8f6d50f2bca4-kube-api-access-z5lgv" (OuterVolumeSpecName: "kube-api-access-z5lgv") pod "08daeede-e190-4e11-a04c-8f6d50f2bca4" (UID: "08daeede-e190-4e11-a04c-8f6d50f2bca4"). InnerVolumeSpecName "kube-api-access-z5lgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:34:11.470143 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.470123 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08daeede-e190-4e11-a04c-8f6d50f2bca4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "08daeede-e190-4e11-a04c-8f6d50f2bca4" (UID: "08daeede-e190-4e11-a04c-8f6d50f2bca4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:34:11.569129 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.569088 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5lgv\" (UniqueName: \"kubernetes.io/projected/08daeede-e190-4e11-a04c-8f6d50f2bca4-kube-api-access-z5lgv\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:34:11.569129 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.569124 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08daeede-e190-4e11-a04c-8f6d50f2bca4-openshift-service-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:34:11.569129 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.569137 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08daeede-e190-4e11-a04c-8f6d50f2bca4-maas-api-tls\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:34:11.984701 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.984669 2560 generic.go:356] "Generic (PLEG): container finished" podID="08daeede-e190-4e11-a04c-8f6d50f2bca4" containerID="01a0bb1b180c0facb1fa69af3cb202959b7d3fbab4fc8b16cdb8c9a27aa5db8e" exitCode=0 Jul 07 14:34:11.985200 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.984714 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" Jul 07 14:34:11.985200 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.984758 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" event={"ID":"08daeede-e190-4e11-a04c-8f6d50f2bca4","Type":"ContainerDied","Data":"01a0bb1b180c0facb1fa69af3cb202959b7d3fbab4fc8b16cdb8c9a27aa5db8e"} Jul 07 14:34:11.985200 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.984819 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2" event={"ID":"08daeede-e190-4e11-a04c-8f6d50f2bca4","Type":"ContainerDied","Data":"14dcad91d1d4801c5106f0f23a84fe33a2aa086e90a84adce695b603dd97faec"} Jul 07 14:34:11.985200 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.984856 2560 scope.go:117] "RemoveContainer" containerID="01a0bb1b180c0facb1fa69af3cb202959b7d3fbab4fc8b16cdb8c9a27aa5db8e" Jul 07 14:34:11.994684 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.994664 2560 scope.go:117] "RemoveContainer" containerID="01a0bb1b180c0facb1fa69af3cb202959b7d3fbab4fc8b16cdb8c9a27aa5db8e" Jul 07 14:34:11.994922 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:34:11.994905 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a0bb1b180c0facb1fa69af3cb202959b7d3fbab4fc8b16cdb8c9a27aa5db8e\": container with ID starting with 01a0bb1b180c0facb1fa69af3cb202959b7d3fbab4fc8b16cdb8c9a27aa5db8e not found: ID does not exist" containerID="01a0bb1b180c0facb1fa69af3cb202959b7d3fbab4fc8b16cdb8c9a27aa5db8e" Jul 07 14:34:11.994976 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:11.994929 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a0bb1b180c0facb1fa69af3cb202959b7d3fbab4fc8b16cdb8c9a27aa5db8e"} err="failed to get container status \"01a0bb1b180c0facb1fa69af3cb202959b7d3fbab4fc8b16cdb8c9a27aa5db8e\": rpc error: code = NotFound desc = could not find container \"01a0bb1b180c0facb1fa69af3cb202959b7d3fbab4fc8b16cdb8c9a27aa5db8e\": container with ID starting with 01a0bb1b180c0facb1fa69af3cb202959b7d3fbab4fc8b16cdb8c9a27aa5db8e not found: ID does not exist" Jul 07 14:34:12.005626 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:12.005594 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2"] Jul 07 14:34:12.008979 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:12.008958 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-42700fcd-5ffb47558-jdvf2"] Jul 07 14:34:12.210684 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:12.210650 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08daeede-e190-4e11-a04c-8f6d50f2bca4" path="/var/lib/kubelet/pods/08daeede-e190-4e11-a04c-8f6d50f2bca4/volumes" Jul 07 14:34:16.891899 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:16.891867 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:16.993374 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:16.993348 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:19.987671 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:19.987633 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:22.614828 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:22.614792 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:22.676180 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:22.676152 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:22.800195 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:22.800163 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:22.899960 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:22.899874 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:23.891873 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:23.891844 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:23.965824 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:23.965788 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q"] Jul 07 14:34:23.966144 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:23.966132 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08daeede-e190-4e11-a04c-8f6d50f2bca4" containerName="maas-api" Jul 07 14:34:23.966196 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:23.966146 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="08daeede-e190-4e11-a04c-8f6d50f2bca4" containerName="maas-api" Jul 07 14:34:23.966233 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:23.966207 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="08daeede-e190-4e11-a04c-8f6d50f2bca4" containerName="maas-api" Jul 07 14:34:23.970489 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:23.970470 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" Jul 07 14:34:23.973674 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:23.973658 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-4b93c8a3-serving-cert\"" Jul 07 14:34:23.980782 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:23.980762 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q"] Jul 07 14:34:23.984096 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:23.984075 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:24.063286 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:24.063238 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:24.083475 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:24.083445 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-maas-api-tls\") pod \"maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q\" (UID: \"e53ecae5-9274-44ea-abdf-ba3906d6f2ae\") " pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" Jul 07 14:34:24.083475 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:24.083477 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqrm6\" (UniqueName: \"kubernetes.io/projected/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-kube-api-access-vqrm6\") pod \"maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q\" (UID: \"e53ecae5-9274-44ea-abdf-ba3906d6f2ae\") " pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" Jul 07 14:34:24.083681 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:24.083508 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-openshift-service-ca\") pod \"maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q\" (UID: \"e53ecae5-9274-44ea-abdf-ba3906d6f2ae\") " pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" Jul 07 14:34:24.184494 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:24.184400 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-maas-api-tls\") pod \"maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q\" (UID: \"e53ecae5-9274-44ea-abdf-ba3906d6f2ae\") " pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" Jul 07 14:34:24.184494 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:24.184436 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqrm6\" (UniqueName: \"kubernetes.io/projected/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-kube-api-access-vqrm6\") pod \"maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q\" (UID: \"e53ecae5-9274-44ea-abdf-ba3906d6f2ae\") " pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" Jul 07 14:34:24.184494 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:24.184467 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-openshift-service-ca\") pod \"maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q\" (UID: \"e53ecae5-9274-44ea-abdf-ba3906d6f2ae\") " pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" Jul 07 14:34:24.185187 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:24.185155 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-openshift-service-ca\") pod \"maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q\" (UID: \"e53ecae5-9274-44ea-abdf-ba3906d6f2ae\") " pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" Jul 07 14:34:24.187036 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:24.187015 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-maas-api-tls\") pod \"maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q\" (UID: \"e53ecae5-9274-44ea-abdf-ba3906d6f2ae\") " pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" Jul 07 14:34:24.192658 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:24.192632 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqrm6\" (UniqueName: \"kubernetes.io/projected/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-kube-api-access-vqrm6\") pod \"maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q\" (UID: \"e53ecae5-9274-44ea-abdf-ba3906d6f2ae\") " pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" Jul 07 14:34:24.213314 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:24.213256 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:24.281835 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:24.281767 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" Jul 07 14:34:24.423303 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:24.420651 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q"] Jul 07 14:34:24.627732 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:24.627697 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:24.728716 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:24.728684 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:25.059098 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:25.059068 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" event={"ID":"e53ecae5-9274-44ea-abdf-ba3906d6f2ae","Type":"ContainerStarted","Data":"239bab662f814d8849416c5dc6fc3e676a37325479e292e833bc7d575b45dab2"} Jul 07 14:34:26.065060 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:26.065025 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" event={"ID":"e53ecae5-9274-44ea-abdf-ba3906d6f2ae","Type":"ContainerStarted","Data":"ec86fd7aa928ea1f7699217d1e8e9aab60f9fe174181feb9ae34035743b7bbe5"} Jul 07 14:34:26.065587 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:26.065186 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" Jul 07 14:34:26.081607 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:26.081531 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" podStartSLOduration=2.466752506 podStartE2EDuration="3.081509893s" totalImagesPullingTime="614.757387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:26.080002592 +0000 UTC m=+1624.512245094" watchObservedRunningTime="2026-07-07 14:34:26.081509893 +0000 UTC m=+1624.513752396" Jul 07 14:34:26.214665 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:26.214633 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:28.324663 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:28.324634 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:28.525234 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:28.525192 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:28.626310 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:28.626210 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q"] Jul 07 14:34:28.626473 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:28.626436 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" podUID="e53ecae5-9274-44ea-abdf-ba3906d6f2ae" containerName="maas-api" containerID="cri-o://ec86fd7aa928ea1f7699217d1e8e9aab60f9fe174181feb9ae34035743b7bbe5" gracePeriod=30 Jul 07 14:34:28.627707 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:28.627683 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:28.682792 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:28.682766 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:28.877961 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:28.877900 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" Jul 07 14:34:28.933568 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:28.933538 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-maas-api-tls\") pod \"e53ecae5-9274-44ea-abdf-ba3906d6f2ae\" (UID: \"e53ecae5-9274-44ea-abdf-ba3906d6f2ae\") " Jul 07 14:34:28.933767 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:28.933622 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqrm6\" (UniqueName: \"kubernetes.io/projected/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-kube-api-access-vqrm6\") pod \"e53ecae5-9274-44ea-abdf-ba3906d6f2ae\" (UID: \"e53ecae5-9274-44ea-abdf-ba3906d6f2ae\") " Jul 07 14:34:28.933767 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:28.933693 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-openshift-service-ca\") pod \"e53ecae5-9274-44ea-abdf-ba3906d6f2ae\" (UID: \"e53ecae5-9274-44ea-abdf-ba3906d6f2ae\") " Jul 07 14:34:28.934076 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:28.934053 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e53ecae5-9274-44ea-abdf-ba3906d6f2ae" (UID: "e53ecae5-9274-44ea-abdf-ba3906d6f2ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:34:28.935815 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:28.935789 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e53ecae5-9274-44ea-abdf-ba3906d6f2ae" (UID: "e53ecae5-9274-44ea-abdf-ba3906d6f2ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:34:28.935903 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:28.935827 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-kube-api-access-vqrm6" (OuterVolumeSpecName: "kube-api-access-vqrm6") pod "e53ecae5-9274-44ea-abdf-ba3906d6f2ae" (UID: "e53ecae5-9274-44ea-abdf-ba3906d6f2ae"). InnerVolumeSpecName "kube-api-access-vqrm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:34:29.034461 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:29.034430 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-maas-api-tls\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:34:29.034461 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:29.034454 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqrm6\" (UniqueName: \"kubernetes.io/projected/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-kube-api-access-vqrm6\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:34:29.034461 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:29.034465 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e53ecae5-9274-44ea-abdf-ba3906d6f2ae-openshift-service-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:34:29.080819 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:29.080778 2560 generic.go:356] "Generic (PLEG): container finished" podID="e53ecae5-9274-44ea-abdf-ba3906d6f2ae" containerID="ec86fd7aa928ea1f7699217d1e8e9aab60f9fe174181feb9ae34035743b7bbe5" exitCode=0 Jul 07 14:34:29.080819 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:29.080816 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" Jul 07 14:34:29.081022 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:29.080868 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" event={"ID":"e53ecae5-9274-44ea-abdf-ba3906d6f2ae","Type":"ContainerDied","Data":"ec86fd7aa928ea1f7699217d1e8e9aab60f9fe174181feb9ae34035743b7bbe5"} Jul 07 14:34:29.081022 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:29.080921 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q" event={"ID":"e53ecae5-9274-44ea-abdf-ba3906d6f2ae","Type":"ContainerDied","Data":"239bab662f814d8849416c5dc6fc3e676a37325479e292e833bc7d575b45dab2"} Jul 07 14:34:29.081022 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:29.080943 2560 scope.go:117] "RemoveContainer" containerID="ec86fd7aa928ea1f7699217d1e8e9aab60f9fe174181feb9ae34035743b7bbe5" Jul 07 14:34:29.092132 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:29.092111 2560 scope.go:117] "RemoveContainer" containerID="ec86fd7aa928ea1f7699217d1e8e9aab60f9fe174181feb9ae34035743b7bbe5" Jul 07 14:34:29.092448 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:34:29.092419 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec86fd7aa928ea1f7699217d1e8e9aab60f9fe174181feb9ae34035743b7bbe5\": container with ID starting with ec86fd7aa928ea1f7699217d1e8e9aab60f9fe174181feb9ae34035743b7bbe5 not found: ID does not exist" containerID="ec86fd7aa928ea1f7699217d1e8e9aab60f9fe174181feb9ae34035743b7bbe5" Jul 07 14:34:29.092522 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:29.092464 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec86fd7aa928ea1f7699217d1e8e9aab60f9fe174181feb9ae34035743b7bbe5"} err="failed to get container status \"ec86fd7aa928ea1f7699217d1e8e9aab60f9fe174181feb9ae34035743b7bbe5\": rpc error: code = NotFound desc = could not find container \"ec86fd7aa928ea1f7699217d1e8e9aab60f9fe174181feb9ae34035743b7bbe5\": container with ID starting with ec86fd7aa928ea1f7699217d1e8e9aab60f9fe174181feb9ae34035743b7bbe5 not found: ID does not exist" Jul 07 14:34:29.101142 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:29.101118 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q"] Jul 07 14:34:29.104866 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:29.104839 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-4b93c8a3-6f5fc6fb58-2ch8q"] Jul 07 14:34:29.839554 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:29.839524 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:30.209976 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:30.209894 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e53ecae5-9274-44ea-abdf-ba3906d6f2ae" path="/var/lib/kubelet/pods/e53ecae5-9274-44ea-abdf-ba3906d6f2ae/volumes" Jul 07 14:34:34.394292 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:34.394245 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:34.453393 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:34.453358 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:34.871590 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:34.871555 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:34.964542 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:34.964511 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:35.065236 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:35.065206 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:36.405370 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:36.405338 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:41.978647 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:41.978598 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:42.085146 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:42.085112 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:42.179663 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:42.179614 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:42.281069 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:42.280991 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:42.660375 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:42.660340 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:49.852533 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:34:49.852500 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:08.619918 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:08.619882 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:14.331689 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:14.331655 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:14.435284 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:14.435235 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:14.528521 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:14.528482 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:30.382338 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:30.382308 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:36.417496 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:36.417459 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:36.467521 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:36.467489 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:36.610399 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:36.610370 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:36.749857 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:36.749769 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:47.142020 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:47.141935 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:47.956064 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:47.956023 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:53.578966 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:53.578929 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:53.688876 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:53.688846 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:53.778805 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:53.778770 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:53.879858 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:53.879779 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:54.624199 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:54.624165 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:54.723414 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:54.723384 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:54.824844 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:54.824811 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:56.367344 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:35:56.367316 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:02.324532 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:02.324500 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:02.457886 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:02.457851 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:02.980381 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:02.980346 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:08.520302 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:08.520251 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:08.579030 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:08.578997 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:08.781470 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:08.781388 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:09.055716 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:09.055681 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:09.554591 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:09.554562 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:12.446419 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.446385 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:12.511959 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.511925 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf"] Jul 07 14:36:12.512508 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.512492 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e53ecae5-9274-44ea-abdf-ba3906d6f2ae" containerName="maas-api" Jul 07 14:36:12.512564 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.512512 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e53ecae5-9274-44ea-abdf-ba3906d6f2ae" containerName="maas-api" Jul 07 14:36:12.512601 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.512586 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e53ecae5-9274-44ea-abdf-ba3906d6f2ae" containerName="maas-api" Jul 07 14:36:12.515772 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.515754 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:12.518188 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.518168 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-3883e616-serving-cert\"" Jul 07 14:36:12.525905 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.525833 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:12.526315 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.525905 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf"] Jul 07 14:36:12.563888 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.563851 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a423b7af-8296-4997-86ae-15d810545a69-openshift-service-ca\") pod \"maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf\" (UID: \"a423b7af-8296-4997-86ae-15d810545a69\") " pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:12.564095 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.563897 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a423b7af-8296-4997-86ae-15d810545a69-maas-api-tls\") pod \"maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf\" (UID: \"a423b7af-8296-4997-86ae-15d810545a69\") " pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:12.564095 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.564030 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-469pz\" (UniqueName: \"kubernetes.io/projected/a423b7af-8296-4997-86ae-15d810545a69-kube-api-access-469pz\") pod \"maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf\" (UID: \"a423b7af-8296-4997-86ae-15d810545a69\") " pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:12.586610 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.586574 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:12.665100 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.665063 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a423b7af-8296-4997-86ae-15d810545a69-openshift-service-ca\") pod \"maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf\" (UID: \"a423b7af-8296-4997-86ae-15d810545a69\") " pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:12.665335 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.665118 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a423b7af-8296-4997-86ae-15d810545a69-maas-api-tls\") pod \"maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf\" (UID: \"a423b7af-8296-4997-86ae-15d810545a69\") " pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:12.665335 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.665214 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-469pz\" (UniqueName: \"kubernetes.io/projected/a423b7af-8296-4997-86ae-15d810545a69-kube-api-access-469pz\") pod \"maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf\" (UID: \"a423b7af-8296-4997-86ae-15d810545a69\") " pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:12.665335 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:36:12.665322 2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-3883e616-serving-cert: secret "maas-api-e2e-mt-3883e616-serving-cert" not found Jul 07 14:36:12.665530 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:36:12.665400 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a423b7af-8296-4997-86ae-15d810545a69-maas-api-tls podName:a423b7af-8296-4997-86ae-15d810545a69 nodeName:}" failed. No retries permitted until 2026-07-07 14:36:13.165381173 +0000 UTC m=+1731.597623654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a423b7af-8296-4997-86ae-15d810545a69-maas-api-tls") pod "maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" (UID: "a423b7af-8296-4997-86ae-15d810545a69") : secret "maas-api-e2e-mt-3883e616-serving-cert" not found Jul 07 14:36:12.665734 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.665715 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a423b7af-8296-4997-86ae-15d810545a69-openshift-service-ca\") pod \"maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf\" (UID: \"a423b7af-8296-4997-86ae-15d810545a69\") " pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:12.674207 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.674181 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-469pz\" (UniqueName: \"kubernetes.io/projected/a423b7af-8296-4997-86ae-15d810545a69-kube-api-access-469pz\") pod \"maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf\" (UID: \"a423b7af-8296-4997-86ae-15d810545a69\") " pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:12.896849 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:12.896812 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:13.087122 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:13.087083 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:13.170430 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:13.170341 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a423b7af-8296-4997-86ae-15d810545a69-maas-api-tls\") pod \"maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf\" (UID: \"a423b7af-8296-4997-86ae-15d810545a69\") " pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:13.173076 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:13.173053 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a423b7af-8296-4997-86ae-15d810545a69-maas-api-tls\") pod \"maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf\" (UID: \"a423b7af-8296-4997-86ae-15d810545a69\") " pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:13.187928 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:13.187900 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:13.428664 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:13.428570 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:13.565959 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:13.565920 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf"] Jul 07 14:36:13.568537 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:36:13.568509 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda423b7af_8296_4997_86ae_15d810545a69.slice/crio-2214119011dadad8e1a313f3eefd16a043cc782b288a6177b522af1de9f69d8f WatchSource:0}: Error finding container 2214119011dadad8e1a313f3eefd16a043cc782b288a6177b522af1de9f69d8f: Status 404 returned error can't find the container with id 2214119011dadad8e1a313f3eefd16a043cc782b288a6177b522af1de9f69d8f Jul 07 14:36:13.637990 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:13.637951 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" event={"ID":"a423b7af-8296-4997-86ae-15d810545a69","Type":"ContainerStarted","Data":"2214119011dadad8e1a313f3eefd16a043cc782b288a6177b522af1de9f69d8f"} Jul 07 14:36:14.645099 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:14.645063 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" event={"ID":"a423b7af-8296-4997-86ae-15d810545a69","Type":"ContainerStarted","Data":"b8bb60149a4bf0eb0f12163dc3c744f0e6086b34822af81052f5a0aea116d1b8"} Jul 07 14:36:14.645528 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:14.645185 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:14.661612 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:14.661566 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" podStartSLOduration=2.108756836 podStartE2EDuration="2.661551411s" totalImagesPullingTime="552.794575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:36:14.660278758 +0000 UTC m=+1733.092521250" watchObservedRunningTime="2026-07-07 14:36:14.661551411 +0000 UTC m=+1733.093793904" Jul 07 14:36:16.925567 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:16.925535 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:17.558081 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:17.558051 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:17.658132 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:17.658099 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:17.977938 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:17.977856 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:20.655752 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:20.655719 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:23.731535 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:23.731510 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:23.833425 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:23.833370 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:24.123733 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.123704 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:24.404602 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.404506 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf"] Jul 07 14:36:24.404775 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.404725 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" podUID="a423b7af-8296-4997-86ae-15d810545a69" containerName="maas-api" containerID="cri-o://b8bb60149a4bf0eb0f12163dc3c744f0e6086b34822af81052f5a0aea116d1b8" gracePeriod=30 Jul 07 14:36:24.433258 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.433232 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:24.495375 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.495344 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:24.653438 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.653413 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:24.710341 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.710224 2560 generic.go:356] "Generic (PLEG): container finished" podID="a423b7af-8296-4997-86ae-15d810545a69" containerID="b8bb60149a4bf0eb0f12163dc3c744f0e6086b34822af81052f5a0aea116d1b8" exitCode=0 Jul 07 14:36:24.710341 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.710296 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" Jul 07 14:36:24.710544 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.710330 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" event={"ID":"a423b7af-8296-4997-86ae-15d810545a69","Type":"ContainerDied","Data":"b8bb60149a4bf0eb0f12163dc3c744f0e6086b34822af81052f5a0aea116d1b8"} Jul 07 14:36:24.710544 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.710411 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf" event={"ID":"a423b7af-8296-4997-86ae-15d810545a69","Type":"ContainerDied","Data":"2214119011dadad8e1a313f3eefd16a043cc782b288a6177b522af1de9f69d8f"} Jul 07 14:36:24.710544 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.710437 2560 scope.go:117] "RemoveContainer" containerID="b8bb60149a4bf0eb0f12163dc3c744f0e6086b34822af81052f5a0aea116d1b8" Jul 07 14:36:24.719308 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.719291 2560 scope.go:117] "RemoveContainer" containerID="b8bb60149a4bf0eb0f12163dc3c744f0e6086b34822af81052f5a0aea116d1b8" Jul 07 14:36:24.719562 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:36:24.719545 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8bb60149a4bf0eb0f12163dc3c744f0e6086b34822af81052f5a0aea116d1b8\": container with ID starting with b8bb60149a4bf0eb0f12163dc3c744f0e6086b34822af81052f5a0aea116d1b8 not found: ID does not exist" containerID="b8bb60149a4bf0eb0f12163dc3c744f0e6086b34822af81052f5a0aea116d1b8" Jul 07 14:36:24.719623 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.719569 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8bb60149a4bf0eb0f12163dc3c744f0e6086b34822af81052f5a0aea116d1b8"} err="failed to get container status \"b8bb60149a4bf0eb0f12163dc3c744f0e6086b34822af81052f5a0aea116d1b8\": rpc error: code = NotFound desc = could not find container \"b8bb60149a4bf0eb0f12163dc3c744f0e6086b34822af81052f5a0aea116d1b8\": container with ID starting with b8bb60149a4bf0eb0f12163dc3c744f0e6086b34822af81052f5a0aea116d1b8 not found: ID does not exist" Jul 07 14:36:24.780206 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.780167 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a423b7af-8296-4997-86ae-15d810545a69-maas-api-tls\") pod \"a423b7af-8296-4997-86ae-15d810545a69\" (UID: \"a423b7af-8296-4997-86ae-15d810545a69\") " Jul 07 14:36:24.780626 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.780214 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a423b7af-8296-4997-86ae-15d810545a69-openshift-service-ca\") pod \"a423b7af-8296-4997-86ae-15d810545a69\" (UID: \"a423b7af-8296-4997-86ae-15d810545a69\") " Jul 07 14:36:24.780626 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.780251 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-469pz\" (UniqueName: \"kubernetes.io/projected/a423b7af-8296-4997-86ae-15d810545a69-kube-api-access-469pz\") pod \"a423b7af-8296-4997-86ae-15d810545a69\" (UID: \"a423b7af-8296-4997-86ae-15d810545a69\") " Jul 07 14:36:24.780704 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.780621 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a423b7af-8296-4997-86ae-15d810545a69-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a423b7af-8296-4997-86ae-15d810545a69" (UID: "a423b7af-8296-4997-86ae-15d810545a69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:36:24.782552 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.782524 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a423b7af-8296-4997-86ae-15d810545a69-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a423b7af-8296-4997-86ae-15d810545a69" (UID: "a423b7af-8296-4997-86ae-15d810545a69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:36:24.782687 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.782618 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a423b7af-8296-4997-86ae-15d810545a69-kube-api-access-469pz" (OuterVolumeSpecName: "kube-api-access-469pz") pod "a423b7af-8296-4997-86ae-15d810545a69" (UID: "a423b7af-8296-4997-86ae-15d810545a69"). InnerVolumeSpecName "kube-api-access-469pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:36:24.880996 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.880952 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a423b7af-8296-4997-86ae-15d810545a69-openshift-service-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:36:24.880996 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.880987 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-469pz\" (UniqueName: \"kubernetes.io/projected/a423b7af-8296-4997-86ae-15d810545a69-kube-api-access-469pz\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:36:24.880996 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:24.881001 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a423b7af-8296-4997-86ae-15d810545a69-maas-api-tls\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:36:25.037827 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:25.037795 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf"] Jul 07 14:36:25.039848 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:25.039827 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-3883e616-5f4d7b489-mwqsf"] Jul 07 14:36:26.209661 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:26.209622 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a423b7af-8296-4997-86ae-15d810545a69" path="/var/lib/kubelet/pods/a423b7af-8296-4997-86ae-15d810545a69/volumes" Jul 07 14:36:30.233846 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:30.233782 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:30.335112 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:30.335073 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:30.486809 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:30.486728 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:30.703332 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:30.703294 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:30.823014 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:30.822961 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:30.920473 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:30.920403 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:31.032302 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:31.032220 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:36.817884 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:36.817838 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:36.925127 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:36.925021 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:37.018336 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:37.018301 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:37.148943 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:37.148855 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:42.843756 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:42.843715 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:42.945171 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:42.945136 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:51.667690 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:51.667652 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:51.945842 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:51.945760 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:52.415291 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:52.415237 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:54.643654 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.643621 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:54.714198 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.714166 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc"] Jul 07 14:36:54.714569 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.714555 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a423b7af-8296-4997-86ae-15d810545a69" containerName="maas-api" Jul 07 14:36:54.714618 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.714571 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a423b7af-8296-4997-86ae-15d810545a69" containerName="maas-api" Jul 07 14:36:54.714653 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.714638 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a423b7af-8296-4997-86ae-15d810545a69" containerName="maas-api" Jul 07 14:36:54.717761 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.717742 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:36:54.719852 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.719827 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-184c55e7-serving-cert\"" Jul 07 14:36:54.728330 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.728298 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc"] Jul 07 14:36:54.731626 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.731597 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:54.757996 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.757965 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d83bf1-3a78-4cde-b76d-c6115738344c-maas-api-tls\") pod \"maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc\" (UID: \"32d83bf1-3a78-4cde-b76d-c6115738344c\") " pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:36:54.758141 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.758006 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg62l\" (UniqueName: \"kubernetes.io/projected/32d83bf1-3a78-4cde-b76d-c6115738344c-kube-api-access-sg62l\") pod \"maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc\" (UID: \"32d83bf1-3a78-4cde-b76d-c6115738344c\") " pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:36:54.758141 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.758048 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d83bf1-3a78-4cde-b76d-c6115738344c-openshift-service-ca\") pod \"maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc\" (UID: \"32d83bf1-3a78-4cde-b76d-c6115738344c\") " pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:36:54.791336 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.791299 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:54.859484 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.859445 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d83bf1-3a78-4cde-b76d-c6115738344c-maas-api-tls\") pod \"maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc\" (UID: \"32d83bf1-3a78-4cde-b76d-c6115738344c\") " pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:36:54.859664 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.859501 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg62l\" (UniqueName: \"kubernetes.io/projected/32d83bf1-3a78-4cde-b76d-c6115738344c-kube-api-access-sg62l\") pod \"maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc\" (UID: \"32d83bf1-3a78-4cde-b76d-c6115738344c\") " pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:36:54.859664 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.859543 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d83bf1-3a78-4cde-b76d-c6115738344c-openshift-service-ca\") pod \"maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc\" (UID: \"32d83bf1-3a78-4cde-b76d-c6115738344c\") " pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:36:54.859664 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:36:54.859601 2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-184c55e7-serving-cert: secret "maas-api-e2e-mt-184c55e7-serving-cert" not found Jul 07 14:36:54.859786 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:36:54.859677 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d83bf1-3a78-4cde-b76d-c6115738344c-maas-api-tls podName:32d83bf1-3a78-4cde-b76d-c6115738344c nodeName:}" failed. No retries permitted until 2026-07-07 14:36:55.359661248 +0000 UTC m=+1773.791903732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32d83bf1-3a78-4cde-b76d-c6115738344c-maas-api-tls") pod "maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" (UID: "32d83bf1-3a78-4cde-b76d-c6115738344c") : secret "maas-api-e2e-mt-184c55e7-serving-cert" not found Jul 07 14:36:54.860182 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.860162 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d83bf1-3a78-4cde-b76d-c6115738344c-openshift-service-ca\") pod \"maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc\" (UID: \"32d83bf1-3a78-4cde-b76d-c6115738344c\") " pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:36:54.869537 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:54.869512 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg62l\" (UniqueName: \"kubernetes.io/projected/32d83bf1-3a78-4cde-b76d-c6115738344c-kube-api-access-sg62l\") pod \"maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc\" (UID: \"32d83bf1-3a78-4cde-b76d-c6115738344c\") " pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:36:55.300013 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:55.299984 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:55.366068 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:55.366030 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d83bf1-3a78-4cde-b76d-c6115738344c-maas-api-tls\") pod \"maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc\" (UID: \"32d83bf1-3a78-4cde-b76d-c6115738344c\") " pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:36:55.368682 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:55.368662 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d83bf1-3a78-4cde-b76d-c6115738344c-maas-api-tls\") pod \"maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc\" (UID: \"32d83bf1-3a78-4cde-b76d-c6115738344c\") " pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:36:55.400234 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:55.400197 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:55.629762 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:55.629676 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:36:55.766092 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:55.766067 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc"] Jul 07 14:36:55.768149 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:36:55.768121 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d83bf1_3a78_4cde_b76d_c6115738344c.slice/crio-f8ef63a3e96ad39ae86a7a7992d454dde8aed8bd5fb63389c25a1db1a126abde WatchSource:0}: Error finding container f8ef63a3e96ad39ae86a7a7992d454dde8aed8bd5fb63389c25a1db1a126abde: Status 404 returned error can't find the container with id f8ef63a3e96ad39ae86a7a7992d454dde8aed8bd5fb63389c25a1db1a126abde Jul 07 14:36:55.884823 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:55.884738 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" event={"ID":"32d83bf1-3a78-4cde-b76d-c6115738344c","Type":"ContainerStarted","Data":"f8ef63a3e96ad39ae86a7a7992d454dde8aed8bd5fb63389c25a1db1a126abde"} Jul 07 14:36:56.892243 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:56.892207 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" event={"ID":"32d83bf1-3a78-4cde-b76d-c6115738344c","Type":"ContainerStarted","Data":"63d550885a8cf9bd811018730a49cdf927fc13856075d2e5d8693e266384e741"} Jul 07 14:36:56.892748 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:56.892324 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:36:56.909331 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:56.909282 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" podStartSLOduration=2.617107893 podStartE2EDuration="2.909251123s" totalImagesPullingTime="292.14323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:36:56.907219455 +0000 UTC m=+1775.339461959" watchObservedRunningTime="2026-07-07 14:36:56.909251123 +0000 UTC m=+1775.341493626" Jul 07 14:36:59.122951 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:36:59.122923 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:37:02.906317 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:02.906291 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:37:25.505878 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:25.505836 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:37:41.335078 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:41.335038 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:37:41.436746 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:41.436706 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:37:47.053544 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:47.053512 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:37:47.463513 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:47.463429 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:37:47.560215 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:47.560180 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:37:47.725616 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:47.725525 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc"] Jul 07 14:37:47.725783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:47.725756 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" podUID="32d83bf1-3a78-4cde-b76d-c6115738344c" containerName="maas-api" containerID="cri-o://63d550885a8cf9bd811018730a49cdf927fc13856075d2e5d8693e266384e741" gracePeriod=30 Jul 07 14:37:47.760911 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:47.760887 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:37:47.977733 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:47.977675 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:37:48.053648 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.053612 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg62l\" (UniqueName: \"kubernetes.io/projected/32d83bf1-3a78-4cde-b76d-c6115738344c-kube-api-access-sg62l\") pod \"32d83bf1-3a78-4cde-b76d-c6115738344c\" (UID: \"32d83bf1-3a78-4cde-b76d-c6115738344c\") " Jul 07 14:37:48.054097 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.053680 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d83bf1-3a78-4cde-b76d-c6115738344c-openshift-service-ca\") pod \"32d83bf1-3a78-4cde-b76d-c6115738344c\" (UID: \"32d83bf1-3a78-4cde-b76d-c6115738344c\") " Jul 07 14:37:48.054097 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.053750 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d83bf1-3a78-4cde-b76d-c6115738344c-maas-api-tls\") pod \"32d83bf1-3a78-4cde-b76d-c6115738344c\" (UID: \"32d83bf1-3a78-4cde-b76d-c6115738344c\") " Jul 07 14:37:48.054097 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.054003 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d83bf1-3a78-4cde-b76d-c6115738344c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32d83bf1-3a78-4cde-b76d-c6115738344c" (UID: "32d83bf1-3a78-4cde-b76d-c6115738344c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:37:48.055867 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.055845 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d83bf1-3a78-4cde-b76d-c6115738344c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32d83bf1-3a78-4cde-b76d-c6115738344c" (UID: "32d83bf1-3a78-4cde-b76d-c6115738344c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:37:48.055942 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.055846 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d83bf1-3a78-4cde-b76d-c6115738344c-kube-api-access-sg62l" (OuterVolumeSpecName: "kube-api-access-sg62l") pod "32d83bf1-3a78-4cde-b76d-c6115738344c" (UID: "32d83bf1-3a78-4cde-b76d-c6115738344c"). InnerVolumeSpecName "kube-api-access-sg62l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:37:48.154503 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.154466 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d83bf1-3a78-4cde-b76d-c6115738344c-maas-api-tls\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:37:48.154503 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.154499 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg62l\" (UniqueName: \"kubernetes.io/projected/32d83bf1-3a78-4cde-b76d-c6115738344c-kube-api-access-sg62l\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:37:48.154503 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.154513 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d83bf1-3a78-4cde-b76d-c6115738344c-openshift-service-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:37:48.165141 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.165105 2560 generic.go:356] "Generic (PLEG): container finished" podID="32d83bf1-3a78-4cde-b76d-c6115738344c" containerID="63d550885a8cf9bd811018730a49cdf927fc13856075d2e5d8693e266384e741" exitCode=0 Jul 07 14:37:48.165334 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.165154 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" Jul 07 14:37:48.165334 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.165172 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" event={"ID":"32d83bf1-3a78-4cde-b76d-c6115738344c","Type":"ContainerDied","Data":"63d550885a8cf9bd811018730a49cdf927fc13856075d2e5d8693e266384e741"} Jul 07 14:37:48.165334 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.165216 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" event={"ID":"32d83bf1-3a78-4cde-b76d-c6115738344c","Type":"ContainerDied","Data":"f8ef63a3e96ad39ae86a7a7992d454dde8aed8bd5fb63389c25a1db1a126abde"} Jul 07 14:37:48.165334 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.165237 2560 scope.go:117] "RemoveContainer" containerID="63d550885a8cf9bd811018730a49cdf927fc13856075d2e5d8693e266384e741" Jul 07 14:37:48.174804 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.174785 2560 scope.go:117] "RemoveContainer" containerID="63d550885a8cf9bd811018730a49cdf927fc13856075d2e5d8693e266384e741" Jul 07 14:37:48.175077 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:37:48.175057 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d550885a8cf9bd811018730a49cdf927fc13856075d2e5d8693e266384e741\": container with ID starting with 63d550885a8cf9bd811018730a49cdf927fc13856075d2e5d8693e266384e741 not found: ID does not exist" containerID="63d550885a8cf9bd811018730a49cdf927fc13856075d2e5d8693e266384e741" Jul 07 14:37:48.175155 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.175089 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d550885a8cf9bd811018730a49cdf927fc13856075d2e5d8693e266384e741"} err="failed to get container status \"63d550885a8cf9bd811018730a49cdf927fc13856075d2e5d8693e266384e741\": rpc error: code = NotFound desc = could not find container \"63d550885a8cf9bd811018730a49cdf927fc13856075d2e5d8693e266384e741\": container with ID starting with 63d550885a8cf9bd811018730a49cdf927fc13856075d2e5d8693e266384e741 not found: ID does not exist" Jul 07 14:37:48.185792 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.185765 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc"] Jul 07 14:37:48.189749 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.189727 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc"] Jul 07 14:37:48.208842 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:48.208813 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32d83bf1-3a78-4cde-b76d-c6115738344c" path="/var/lib/kubelet/pods/32d83bf1-3a78-4cde-b76d-c6115738344c/volumes" Jul 07 14:37:50.901520 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:50.901477 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-mt-184c55e7-5c889cbdf-hp5dc" podUID="32d83bf1-3a78-4cde-b76d-c6115738344c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 07 14:37:53.092672 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:53.092634 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:37:58.118800 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:37:58.118765 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:34.248427 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:34.248395 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:35.167776 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:35.167743 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:40.666934 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:40.666904 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:40.767334 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:40.767303 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:40.949318 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:40.949227 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:41.182242 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:41.182212 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:42.213572 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:42.213482 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:42.312448 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:42.312405 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:42.412674 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:42.412633 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:43.086601 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:43.086563 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:43.189877 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:43.189829 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:43.288557 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:43.288524 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:44.618220 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:44.618179 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:50.326211 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:50.326181 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:51.634281 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:51.634241 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:56.526815 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:56.526764 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:56.586042 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:56.586005 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:56.675199 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:56.675165 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:56.772766 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:38:56.772736 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:05.429309 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:05.429236 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:05.530157 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:05.530102 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:07.232661 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:07.232635 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:07.569388 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:07.569355 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:28.518214 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:28.518175 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:34.150699 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:34.150660 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:34.208527 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:34.208498 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:34.508318 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:34.508197 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:34.873167 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:34.873134 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:34.936615 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:34.936579 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:37.446450 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.446418 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:37.545792 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.545760 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:37.656964 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.656932 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw"] Jul 07 14:39:37.657454 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.657428 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d83bf1-3a78-4cde-b76d-c6115738344c" containerName="maas-api" Jul 07 14:39:37.657454 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.657448 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d83bf1-3a78-4cde-b76d-c6115738344c" containerName="maas-api" Jul 07 14:39:37.657652 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.657519 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="32d83bf1-3a78-4cde-b76d-c6115738344c" containerName="maas-api" Jul 07 14:39:37.660506 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.660488 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:39:37.662778 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.662758 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-fd6585-serving-cert\"" Jul 07 14:39:37.671127 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.671106 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw"] Jul 07 14:39:37.686719 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.686694 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:37.793849 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.793826 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5759de82-856b-42e9-afaa-31be7c004316-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw\" (UID: \"5759de82-856b-42e9-afaa-31be7c004316\") " pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:39:37.794016 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.793866 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjghf\" (UniqueName: \"kubernetes.io/projected/5759de82-856b-42e9-afaa-31be7c004316-kube-api-access-rjghf\") pod \"maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw\" (UID: \"5759de82-856b-42e9-afaa-31be7c004316\") " pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:39:37.794016 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.793977 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5759de82-856b-42e9-afaa-31be7c004316-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw\" (UID: \"5759de82-856b-42e9-afaa-31be7c004316\") " pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:39:37.894890 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.894856 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5759de82-856b-42e9-afaa-31be7c004316-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw\" (UID: \"5759de82-856b-42e9-afaa-31be7c004316\") " pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:39:37.895062 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.894901 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjghf\" (UniqueName: \"kubernetes.io/projected/5759de82-856b-42e9-afaa-31be7c004316-kube-api-access-rjghf\") pod \"maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw\" (UID: \"5759de82-856b-42e9-afaa-31be7c004316\") " pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:39:37.895062 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.894944 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5759de82-856b-42e9-afaa-31be7c004316-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw\" (UID: \"5759de82-856b-42e9-afaa-31be7c004316\") " pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:39:37.895134 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:39:37.895059 2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-fd6585-serving-cert: secret "maas-api-e2e-inf-a-fd6585-serving-cert" not found Jul 07 14:39:37.895134 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:39:37.895120 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5759de82-856b-42e9-afaa-31be7c004316-maas-api-tls podName:5759de82-856b-42e9-afaa-31be7c004316 nodeName:}" failed. No retries permitted until 2026-07-07 14:39:38.39510151 +0000 UTC m=+1936.827343999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5759de82-856b-42e9-afaa-31be7c004316-maas-api-tls") pod "maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" (UID: "5759de82-856b-42e9-afaa-31be7c004316") : secret "maas-api-e2e-inf-a-fd6585-serving-cert" not found Jul 07 14:39:37.895604 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.895584 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5759de82-856b-42e9-afaa-31be7c004316-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw\" (UID: \"5759de82-856b-42e9-afaa-31be7c004316\") " pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:39:37.907814 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:37.907787 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjghf\" (UniqueName: \"kubernetes.io/projected/5759de82-856b-42e9-afaa-31be7c004316-kube-api-access-rjghf\") pod \"maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw\" (UID: \"5759de82-856b-42e9-afaa-31be7c004316\") " pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:39:38.092835 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:38.092746 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:38.195499 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:38.195471 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:38.399802 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:38.399710 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5759de82-856b-42e9-afaa-31be7c004316-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw\" (UID: \"5759de82-856b-42e9-afaa-31be7c004316\") " pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:39:38.402397 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:38.402368 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5759de82-856b-42e9-afaa-31be7c004316-maas-api-tls\") pod \"maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw\" (UID: \"5759de82-856b-42e9-afaa-31be7c004316\") " pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:39:38.572971 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:38.572933 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:39:38.709656 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:38.709611 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw"] Jul 07 14:39:38.715653 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:39:38.715627 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5759de82_856b_42e9_afaa_31be7c004316.slice/crio-5f378f4d647e822906d49ad5259fce169518fd29f3ffa9d878181729c7c587c8 WatchSource:0}: Error finding container 5f378f4d647e822906d49ad5259fce169518fd29f3ffa9d878181729c7c587c8: Status 404 returned error can't find the container with id 5f378f4d647e822906d49ad5259fce169518fd29f3ffa9d878181729c7c587c8 Jul 07 14:39:38.716834 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:38.716818 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:39:38.747162 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:38.747130 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" event={"ID":"5759de82-856b-42e9-afaa-31be7c004316","Type":"ContainerStarted","Data":"5f378f4d647e822906d49ad5259fce169518fd29f3ffa9d878181729c7c587c8"} Jul 07 14:39:39.753259 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:39.753221 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" event={"ID":"5759de82-856b-42e9-afaa-31be7c004316","Type":"ContainerStarted","Data":"d7cdea625f015cf68d203120aa64e5f76e6aaaa4cd5b3adcb8a85574360eb196"} Jul 07 14:39:39.753664 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:39.753405 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:39:39.770326 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:39.770279 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" podStartSLOduration=2.276510591 podStartE2EDuration="2.770250843s" totalImagesPullingTime="493.740252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:39.767416821 +0000 UTC m=+1938.199659323" watchObservedRunningTime="2026-07-07 14:39:39.770250843 +0000 UTC m=+1938.202493343" Jul 07 14:39:41.958718 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:41.958680 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:42.311947 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:42.311918 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:42.614566 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:42.614487 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:42.739331 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:42.739293 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:45.763497 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:45.763469 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:39:47.917722 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:47.917682 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:47.961656 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:47.961628 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj"] Jul 07 14:39:47.965189 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:47.965171 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" Jul 07 14:39:47.967528 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:47.967511 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-8d4709-serving-cert\"" Jul 07 14:39:47.978828 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:47.978807 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj"] Jul 07 14:39:47.996906 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:47.996864 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:48.087180 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.087144 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbe5919-365d-4e5b-9557-cbc69b112968-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj\" (UID: \"5cbe5919-365d-4e5b-9557-cbc69b112968\") " pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" Jul 07 14:39:48.087180 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.087181 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r77tm\" (UniqueName: \"kubernetes.io/projected/5cbe5919-365d-4e5b-9557-cbc69b112968-kube-api-access-r77tm\") pod \"maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj\" (UID: \"5cbe5919-365d-4e5b-9557-cbc69b112968\") " pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" Jul 07 14:39:48.087406 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.087340 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cbe5919-365d-4e5b-9557-cbc69b112968-maas-api-tls\") pod \"maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj\" (UID: \"5cbe5919-365d-4e5b-9557-cbc69b112968\") " pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" Jul 07 14:39:48.090238 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.090217 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:48.187864 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.187784 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbe5919-365d-4e5b-9557-cbc69b112968-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj\" (UID: \"5cbe5919-365d-4e5b-9557-cbc69b112968\") " pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" Jul 07 14:39:48.187864 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.187818 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r77tm\" (UniqueName: \"kubernetes.io/projected/5cbe5919-365d-4e5b-9557-cbc69b112968-kube-api-access-r77tm\") pod \"maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj\" (UID: \"5cbe5919-365d-4e5b-9557-cbc69b112968\") " pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" Jul 07 14:39:48.188051 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.187873 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cbe5919-365d-4e5b-9557-cbc69b112968-maas-api-tls\") pod \"maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj\" (UID: \"5cbe5919-365d-4e5b-9557-cbc69b112968\") " pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" Jul 07 14:39:48.188489 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.188461 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbe5919-365d-4e5b-9557-cbc69b112968-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj\" (UID: \"5cbe5919-365d-4e5b-9557-cbc69b112968\") " pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" Jul 07 14:39:48.190389 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.190365 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cbe5919-365d-4e5b-9557-cbc69b112968-maas-api-tls\") pod \"maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj\" (UID: \"5cbe5919-365d-4e5b-9557-cbc69b112968\") " pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" Jul 07 14:39:48.199931 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.199911 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r77tm\" (UniqueName: \"kubernetes.io/projected/5cbe5919-365d-4e5b-9557-cbc69b112968-kube-api-access-r77tm\") pod \"maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj\" (UID: \"5cbe5919-365d-4e5b-9557-cbc69b112968\") " pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" Jul 07 14:39:48.263819 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.263790 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:48.275388 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.275367 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" Jul 07 14:39:48.410428 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.410406 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj"] Jul 07 14:39:48.411963 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:39:48.411937 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbe5919_365d_4e5b_9557_cbc69b112968.slice/crio-cbd727f51e4c0955d57619a319e08c113f489625a85bab85c3d55a9b05f9d545 WatchSource:0}: Error finding container cbd727f51e4c0955d57619a319e08c113f489625a85bab85c3d55a9b05f9d545: Status 404 returned error can't find the container with id cbd727f51e4c0955d57619a319e08c113f489625a85bab85c3d55a9b05f9d545 Jul 07 14:39:48.564861 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.564830 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:48.664633 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.664603 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:48.802147 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:48.802117 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" event={"ID":"5cbe5919-365d-4e5b-9557-cbc69b112968","Type":"ContainerStarted","Data":"cbd727f51e4c0955d57619a319e08c113f489625a85bab85c3d55a9b05f9d545"} Jul 07 14:39:49.809229 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:49.809183 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" event={"ID":"5cbe5919-365d-4e5b-9557-cbc69b112968","Type":"ContainerStarted","Data":"51514d615ba67d37bc9bff703f6a81ea1c3901b5acd752536cb52a341fbdede0"} Jul 07 14:39:49.809633 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:49.809251 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" Jul 07 14:39:49.827093 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:49.827042 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" podStartSLOduration=2.46390959 podStartE2EDuration="2.827025305s" totalImagesPullingTime="363.115715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:39:49.823648098 +0000 UTC m=+1948.255890616" watchObservedRunningTime="2026-07-07 14:39:49.827025305 +0000 UTC m=+1948.259267805" Jul 07 14:39:52.383948 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:52.383921 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:55.819812 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:55.819781 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" Jul 07 14:39:59.595049 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:59.595017 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:59.694936 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:39:59.694908 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:01.315337 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:01.315308 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:09.957161 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:09.957125 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:10.056278 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:10.056241 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:31.074541 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:31.074497 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:50.735187 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:50.735151 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:51.664456 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:51.664419 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:51.831497 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:51.831463 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:52.217240 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:52.217206 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:52.331301 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:52.331246 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:52.488631 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:52.488547 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw"] Jul 07 14:40:52.488785 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:52.488766 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" podUID="5759de82-856b-42e9-afaa-31be7c004316" containerName="maas-api" containerID="cri-o://d7cdea625f015cf68d203120aa64e5f76e6aaaa4cd5b3adcb8a85574360eb196" gracePeriod=30 Jul 07 14:40:52.518374 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:52.518350 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:40:52.743177 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:52.743118 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:40:52.781432 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:52.781405 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjghf\" (UniqueName: \"kubernetes.io/projected/5759de82-856b-42e9-afaa-31be7c004316-kube-api-access-rjghf\") pod \"5759de82-856b-42e9-afaa-31be7c004316\" (UID: \"5759de82-856b-42e9-afaa-31be7c004316\") " Jul 07 14:40:52.781606 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:52.781502 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5759de82-856b-42e9-afaa-31be7c004316-openshift-service-ca\") pod \"5759de82-856b-42e9-afaa-31be7c004316\" (UID: \"5759de82-856b-42e9-afaa-31be7c004316\") " Jul 07 14:40:52.781606 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:52.781525 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5759de82-856b-42e9-afaa-31be7c004316-maas-api-tls\") pod \"5759de82-856b-42e9-afaa-31be7c004316\" (UID: \"5759de82-856b-42e9-afaa-31be7c004316\") " Jul 07 14:40:52.781882 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:52.781861 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5759de82-856b-42e9-afaa-31be7c004316-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5759de82-856b-42e9-afaa-31be7c004316" (UID: "5759de82-856b-42e9-afaa-31be7c004316"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:40:52.783698 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:52.783669 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5759de82-856b-42e9-afaa-31be7c004316-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5759de82-856b-42e9-afaa-31be7c004316" (UID: "5759de82-856b-42e9-afaa-31be7c004316"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:40:52.783809 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:52.783662 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5759de82-856b-42e9-afaa-31be7c004316-kube-api-access-rjghf" (OuterVolumeSpecName: "kube-api-access-rjghf") pod "5759de82-856b-42e9-afaa-31be7c004316" (UID: "5759de82-856b-42e9-afaa-31be7c004316"). InnerVolumeSpecName "kube-api-access-rjghf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:40:52.882658 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:52.882620 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5759de82-856b-42e9-afaa-31be7c004316-openshift-service-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:40:52.882658 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:52.882651 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5759de82-856b-42e9-afaa-31be7c004316-maas-api-tls\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:40:52.882658 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:52.882661 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjghf\" (UniqueName: \"kubernetes.io/projected/5759de82-856b-42e9-afaa-31be7c004316-kube-api-access-rjghf\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:40:53.140929 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:53.140891 2560 generic.go:356] "Generic (PLEG): container finished" podID="5759de82-856b-42e9-afaa-31be7c004316" containerID="d7cdea625f015cf68d203120aa64e5f76e6aaaa4cd5b3adcb8a85574360eb196" exitCode=0 Jul 07 14:40:53.140929 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:53.140936 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" Jul 07 14:40:53.141156 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:53.140971 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" event={"ID":"5759de82-856b-42e9-afaa-31be7c004316","Type":"ContainerDied","Data":"d7cdea625f015cf68d203120aa64e5f76e6aaaa4cd5b3adcb8a85574360eb196"} Jul 07 14:40:53.141156 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:53.141026 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw" event={"ID":"5759de82-856b-42e9-afaa-31be7c004316","Type":"ContainerDied","Data":"5f378f4d647e822906d49ad5259fce169518fd29f3ffa9d878181729c7c587c8"} Jul 07 14:40:53.141156 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:53.141050 2560 scope.go:117] "RemoveContainer" containerID="d7cdea625f015cf68d203120aa64e5f76e6aaaa4cd5b3adcb8a85574360eb196" Jul 07 14:40:53.150303 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:53.150284 2560 scope.go:117] "RemoveContainer" containerID="d7cdea625f015cf68d203120aa64e5f76e6aaaa4cd5b3adcb8a85574360eb196" Jul 07 14:40:53.150565 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:40:53.150548 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7cdea625f015cf68d203120aa64e5f76e6aaaa4cd5b3adcb8a85574360eb196\": container with ID starting with d7cdea625f015cf68d203120aa64e5f76e6aaaa4cd5b3adcb8a85574360eb196 not found: ID does not exist" containerID="d7cdea625f015cf68d203120aa64e5f76e6aaaa4cd5b3adcb8a85574360eb196" Jul 07 14:40:53.150625 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:53.150577 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7cdea625f015cf68d203120aa64e5f76e6aaaa4cd5b3adcb8a85574360eb196"} err="failed to get container status \"d7cdea625f015cf68d203120aa64e5f76e6aaaa4cd5b3adcb8a85574360eb196\": rpc error: code = NotFound desc = could not find container \"d7cdea625f015cf68d203120aa64e5f76e6aaaa4cd5b3adcb8a85574360eb196\": container with ID starting with d7cdea625f015cf68d203120aa64e5f76e6aaaa4cd5b3adcb8a85574360eb196 not found: ID does not exist" Jul 07 14:40:53.161717 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:53.161689 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw"] Jul 07 14:40:53.166789 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:53.166768 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-fd6585-59b77bcdcc-grwmw"] Jul 07 14:40:54.208477 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:54.208441 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5759de82-856b-42e9-afaa-31be7c004316" path="/var/lib/kubelet/pods/5759de82-856b-42e9-afaa-31be7c004316/volumes" Jul 07 14:40:57.299819 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:40:57.299786 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:08.704609 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:08.704560 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:08.757534 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:08.757504 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:33.945470 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:33.945433 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:34.228569 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:34.228479 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:34.445456 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:34.445421 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:34.690348 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:34.690295 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:35.128909 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:35.128875 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:35.229323 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:35.229289 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:35.376515 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:35.376474 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj"] Jul 07 14:41:35.376764 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:35.376734 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" podUID="5cbe5919-365d-4e5b-9557-cbc69b112968" containerName="maas-api" containerID="cri-o://51514d615ba67d37bc9bff703f6a81ea1c3901b5acd752536cb52a341fbdede0" gracePeriod=30 Jul 07 14:41:35.420886 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:35.420855 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:35.628233 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:35.628206 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" Jul 07 14:41:35.667376 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:35.667290 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cbe5919-365d-4e5b-9557-cbc69b112968-maas-api-tls\") pod \"5cbe5919-365d-4e5b-9557-cbc69b112968\" (UID: \"5cbe5919-365d-4e5b-9557-cbc69b112968\") " Jul 07 14:41:35.667538 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:35.667392 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r77tm\" (UniqueName: \"kubernetes.io/projected/5cbe5919-365d-4e5b-9557-cbc69b112968-kube-api-access-r77tm\") pod \"5cbe5919-365d-4e5b-9557-cbc69b112968\" (UID: \"5cbe5919-365d-4e5b-9557-cbc69b112968\") " Jul 07 14:41:35.667608 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:35.667568 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbe5919-365d-4e5b-9557-cbc69b112968-openshift-service-ca\") pod \"5cbe5919-365d-4e5b-9557-cbc69b112968\" (UID: \"5cbe5919-365d-4e5b-9557-cbc69b112968\") " Jul 07 14:41:35.668055 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:35.668021 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbe5919-365d-4e5b-9557-cbc69b112968-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5cbe5919-365d-4e5b-9557-cbc69b112968" (UID: "5cbe5919-365d-4e5b-9557-cbc69b112968"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:41:35.669508 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:35.669481 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbe5919-365d-4e5b-9557-cbc69b112968-kube-api-access-r77tm" (OuterVolumeSpecName: "kube-api-access-r77tm") pod "5cbe5919-365d-4e5b-9557-cbc69b112968" (UID: "5cbe5919-365d-4e5b-9557-cbc69b112968"). InnerVolumeSpecName "kube-api-access-r77tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:41:35.669786 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:35.669763 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbe5919-365d-4e5b-9557-cbc69b112968-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5cbe5919-365d-4e5b-9557-cbc69b112968" (UID: "5cbe5919-365d-4e5b-9557-cbc69b112968"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:41:35.769342 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:35.769302 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cbe5919-365d-4e5b-9557-cbc69b112968-maas-api-tls\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:41:35.769342 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:35.769337 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r77tm\" (UniqueName: \"kubernetes.io/projected/5cbe5919-365d-4e5b-9557-cbc69b112968-kube-api-access-r77tm\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:41:35.769342 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:35.769347 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cbe5919-365d-4e5b-9557-cbc69b112968-openshift-service-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:41:36.375232 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:36.375196 2560 generic.go:356] "Generic (PLEG): container finished" podID="5cbe5919-365d-4e5b-9557-cbc69b112968" containerID="51514d615ba67d37bc9bff703f6a81ea1c3901b5acd752536cb52a341fbdede0" exitCode=0 Jul 07 14:41:36.375636 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:36.375241 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" Jul 07 14:41:36.375636 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:36.375303 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" event={"ID":"5cbe5919-365d-4e5b-9557-cbc69b112968","Type":"ContainerDied","Data":"51514d615ba67d37bc9bff703f6a81ea1c3901b5acd752536cb52a341fbdede0"} Jul 07 14:41:36.375636 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:36.375358 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj" event={"ID":"5cbe5919-365d-4e5b-9557-cbc69b112968","Type":"ContainerDied","Data":"cbd727f51e4c0955d57619a319e08c113f489625a85bab85c3d55a9b05f9d545"} Jul 07 14:41:36.375636 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:36.375385 2560 scope.go:117] "RemoveContainer" containerID="51514d615ba67d37bc9bff703f6a81ea1c3901b5acd752536cb52a341fbdede0" Jul 07 14:41:36.384550 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:36.384358 2560 scope.go:117] "RemoveContainer" containerID="51514d615ba67d37bc9bff703f6a81ea1c3901b5acd752536cb52a341fbdede0" Jul 07 14:41:36.384653 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:41:36.384634 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51514d615ba67d37bc9bff703f6a81ea1c3901b5acd752536cb52a341fbdede0\": container with ID starting with 51514d615ba67d37bc9bff703f6a81ea1c3901b5acd752536cb52a341fbdede0 not found: ID does not exist" containerID="51514d615ba67d37bc9bff703f6a81ea1c3901b5acd752536cb52a341fbdede0" Jul 07 14:41:36.384692 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:36.384662 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51514d615ba67d37bc9bff703f6a81ea1c3901b5acd752536cb52a341fbdede0"} err="failed to get container status \"51514d615ba67d37bc9bff703f6a81ea1c3901b5acd752536cb52a341fbdede0\": rpc error: code = NotFound desc = could not find container \"51514d615ba67d37bc9bff703f6a81ea1c3901b5acd752536cb52a341fbdede0\": container with ID starting with 51514d615ba67d37bc9bff703f6a81ea1c3901b5acd752536cb52a341fbdede0 not found: ID does not exist" Jul 07 14:41:36.392753 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:36.392727 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj"] Jul 07 14:41:36.397440 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:36.397415 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-8d4709-8d8566746-k8dhj"] Jul 07 14:41:38.209187 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:38.209152 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cbe5919-365d-4e5b-9557-cbc69b112968" path="/var/lib/kubelet/pods/5cbe5919-365d-4e5b-9557-cbc69b112968/volumes" Jul 07 14:41:40.836892 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:40.836850 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:40.888719 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:40.888675 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:41.130525 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:41.130413 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:41.356577 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:41.356519 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:41.694812 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:41.694690 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:42.016729 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:42.016617 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:42.382362 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:42.382331 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:44.499783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.499750 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:44.566016 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.565971 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z"] Jul 07 14:41:44.566356 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.566338 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cbe5919-365d-4e5b-9557-cbc69b112968" containerName="maas-api" Jul 07 14:41:44.566356 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.566354 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cbe5919-365d-4e5b-9557-cbc69b112968" containerName="maas-api" Jul 07 14:41:44.566532 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.566368 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5759de82-856b-42e9-afaa-31be7c004316" containerName="maas-api" Jul 07 14:41:44.566532 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.566374 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5759de82-856b-42e9-afaa-31be7c004316" containerName="maas-api" Jul 07 14:41:44.566532 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.566449 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5759de82-856b-42e9-afaa-31be7c004316" containerName="maas-api" Jul 07 14:41:44.566532 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.566456 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cbe5919-365d-4e5b-9557-cbc69b112968" containerName="maas-api" Jul 07 14:41:44.570771 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.570752 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" Jul 07 14:41:44.572978 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.572950 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-cdb540-serving-cert\"" Jul 07 14:41:44.580124 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.580100 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z"] Jul 07 14:41:44.583716 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.583696 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:44.649416 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.649385 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68592e8a-d720-4aa7-97e3-b3d5330cb031-maas-api-tls\") pod \"maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z\" (UID: \"68592e8a-d720-4aa7-97e3-b3d5330cb031\") " pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" Jul 07 14:41:44.649611 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.649428 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68592e8a-d720-4aa7-97e3-b3d5330cb031-openshift-service-ca\") pod \"maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z\" (UID: \"68592e8a-d720-4aa7-97e3-b3d5330cb031\") " pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" Jul 07 14:41:44.649611 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.649546 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x682d\" (UniqueName: \"kubernetes.io/projected/68592e8a-d720-4aa7-97e3-b3d5330cb031-kube-api-access-x682d\") pod \"maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z\" (UID: \"68592e8a-d720-4aa7-97e3-b3d5330cb031\") " pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" Jul 07 14:41:44.650172 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.650153 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:44.750751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.750662 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68592e8a-d720-4aa7-97e3-b3d5330cb031-openshift-service-ca\") pod \"maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z\" (UID: \"68592e8a-d720-4aa7-97e3-b3d5330cb031\") " pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" Jul 07 14:41:44.750751 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.750734 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x682d\" (UniqueName: \"kubernetes.io/projected/68592e8a-d720-4aa7-97e3-b3d5330cb031-kube-api-access-x682d\") pod \"maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z\" (UID: \"68592e8a-d720-4aa7-97e3-b3d5330cb031\") " pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" Jul 07 14:41:44.750935 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.750775 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68592e8a-d720-4aa7-97e3-b3d5330cb031-maas-api-tls\") pod \"maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z\" (UID: \"68592e8a-d720-4aa7-97e3-b3d5330cb031\") " pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" Jul 07 14:41:44.751345 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.751307 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68592e8a-d720-4aa7-97e3-b3d5330cb031-openshift-service-ca\") pod \"maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z\" (UID: \"68592e8a-d720-4aa7-97e3-b3d5330cb031\") " pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" Jul 07 14:41:44.753309 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.753281 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68592e8a-d720-4aa7-97e3-b3d5330cb031-maas-api-tls\") pod \"maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z\" (UID: \"68592e8a-d720-4aa7-97e3-b3d5330cb031\") " pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" Jul 07 14:41:44.759036 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.759001 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x682d\" (UniqueName: \"kubernetes.io/projected/68592e8a-d720-4aa7-97e3-b3d5330cb031-kube-api-access-x682d\") pod \"maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z\" (UID: \"68592e8a-d720-4aa7-97e3-b3d5330cb031\") " pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" Jul 07 14:41:44.782832 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.782804 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:44.882850 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:44.882762 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" Jul 07 14:41:45.019374 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:45.019347 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z"] Jul 07 14:41:45.021024 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:41:45.020993 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68592e8a_d720_4aa7_97e3_b3d5330cb031.slice/crio-647550a452c8d1f8ca1d2b8c4d75a0f205e7467a52a8fdfb86cb779884702147 WatchSource:0}: Error finding container 647550a452c8d1f8ca1d2b8c4d75a0f205e7467a52a8fdfb86cb779884702147: Status 404 returned error can't find the container with id 647550a452c8d1f8ca1d2b8c4d75a0f205e7467a52a8fdfb86cb779884702147 Jul 07 14:41:45.169426 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:45.169394 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:45.270736 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:45.270663 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:45.427329 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:45.427289 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" event={"ID":"68592e8a-d720-4aa7-97e3-b3d5330cb031","Type":"ContainerStarted","Data":"647550a452c8d1f8ca1d2b8c4d75a0f205e7467a52a8fdfb86cb779884702147"} Jul 07 14:41:46.434505 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:46.434466 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" event={"ID":"68592e8a-d720-4aa7-97e3-b3d5330cb031","Type":"ContainerStarted","Data":"7e3f3136c40fcdcb122115e486c1f888b8d5a9864d572d00b69f20e14df0311a"} Jul 07 14:41:46.434902 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:46.434664 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" Jul 07 14:41:46.451761 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:46.451697 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" podStartSLOduration=1.7169303710000001 podStartE2EDuration="2.451681359s" totalImagesPullingTime="734.750988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:41:46.449218895 +0000 UTC m=+2064.881461396" watchObservedRunningTime="2026-07-07 14:41:46.451681359 +0000 UTC m=+2064.883923881" Jul 07 14:41:49.359176 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:49.359146 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:50.215891 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:50.215858 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:50.690167 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:50.690135 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:50.963745 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:50.963673 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:52.445480 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:52.445450 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" Jul 07 14:41:53.610189 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.610154 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:53.688606 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.688560 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:53.694379 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.694349 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn"] Jul 07 14:41:53.698451 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.698428 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" Jul 07 14:41:53.700885 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.700849 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-8c73bb-serving-cert\"" Jul 07 14:41:53.711234 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.711210 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn"] Jul 07 14:41:53.755257 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.755228 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:53.832847 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.832812 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb9e089-eea2-4be4-9892-0a0173733f5a-openshift-service-ca\") pod \"maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn\" (UID: \"eeb9e089-eea2-4be4-9892-0a0173733f5a\") " pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" Jul 07 14:41:53.833023 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.832870 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87vgr\" (UniqueName: \"kubernetes.io/projected/eeb9e089-eea2-4be4-9892-0a0173733f5a-kube-api-access-87vgr\") pod \"maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn\" (UID: \"eeb9e089-eea2-4be4-9892-0a0173733f5a\") " pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" Jul 07 14:41:53.833023 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.832916 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb9e089-eea2-4be4-9892-0a0173733f5a-maas-api-tls\") pod \"maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn\" (UID: \"eeb9e089-eea2-4be4-9892-0a0173733f5a\") " pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" Jul 07 14:41:53.916335 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.916226 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:53.933580 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.933553 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb9e089-eea2-4be4-9892-0a0173733f5a-openshift-service-ca\") pod \"maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn\" (UID: \"eeb9e089-eea2-4be4-9892-0a0173733f5a\") " pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" Jul 07 14:41:53.933695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.933602 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87vgr\" (UniqueName: \"kubernetes.io/projected/eeb9e089-eea2-4be4-9892-0a0173733f5a-kube-api-access-87vgr\") pod \"maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn\" (UID: \"eeb9e089-eea2-4be4-9892-0a0173733f5a\") " pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" Jul 07 14:41:53.933695 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.933632 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb9e089-eea2-4be4-9892-0a0173733f5a-maas-api-tls\") pod \"maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn\" (UID: \"eeb9e089-eea2-4be4-9892-0a0173733f5a\") " pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" Jul 07 14:41:53.934198 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.934176 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb9e089-eea2-4be4-9892-0a0173733f5a-openshift-service-ca\") pod \"maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn\" (UID: \"eeb9e089-eea2-4be4-9892-0a0173733f5a\") " pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" Jul 07 14:41:53.936162 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.936137 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb9e089-eea2-4be4-9892-0a0173733f5a-maas-api-tls\") pod \"maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn\" (UID: \"eeb9e089-eea2-4be4-9892-0a0173733f5a\") " pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" Jul 07 14:41:53.941658 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:53.941636 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87vgr\" (UniqueName: \"kubernetes.io/projected/eeb9e089-eea2-4be4-9892-0a0173733f5a-kube-api-access-87vgr\") pod \"maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn\" (UID: \"eeb9e089-eea2-4be4-9892-0a0173733f5a\") " pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" Jul 07 14:41:54.012300 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:54.012231 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" Jul 07 14:41:54.145705 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:54.145680 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn"] Jul 07 14:41:54.147280 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:41:54.147242 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9e089_eea2_4be4_9892_0a0173733f5a.slice/crio-0a66dae2c4abbef0cc0ad31474f9a782fbbf559e9663d04b0f69a59c28eb9754 WatchSource:0}: Error finding container 0a66dae2c4abbef0cc0ad31474f9a782fbbf559e9663d04b0f69a59c28eb9754: Status 404 returned error can't find the container with id 0a66dae2c4abbef0cc0ad31474f9a782fbbf559e9663d04b0f69a59c28eb9754 Jul 07 14:41:54.264950 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:54.264864 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:54.365736 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:54.365703 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:41:54.477558 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:54.477526 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" event={"ID":"eeb9e089-eea2-4be4-9892-0a0173733f5a","Type":"ContainerStarted","Data":"0a66dae2c4abbef0cc0ad31474f9a782fbbf559e9663d04b0f69a59c28eb9754"} Jul 07 14:41:55.484446 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:55.484410 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" event={"ID":"eeb9e089-eea2-4be4-9892-0a0173733f5a","Type":"ContainerStarted","Data":"14a0aee70a69e532f99ae42922199a491fd21a10767955da26eb7466ee4d93c9"} Jul 07 14:41:55.484832 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:55.484562 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" Jul 07 14:41:55.502438 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:55.502380 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" podStartSLOduration=1.8171631499999998 podStartE2EDuration="2.502362181s" totalImagesPullingTime="685.199031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:41:55.501822719 +0000 UTC m=+2073.934065229" watchObservedRunningTime="2026-07-07 14:41:55.502362181 +0000 UTC m=+2073.934604685" Jul 07 14:41:58.208644 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:41:58.208594 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:01.495125 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:01.495095 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" Jul 07 14:42:06.560820 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:06.560766 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:06.716415 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:06.716372 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:06.817390 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:06.817309 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:06.935652 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:06.935621 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z"] Jul 07 14:42:06.935871 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:06.935828 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" podUID="68592e8a-d720-4aa7-97e3-b3d5330cb031" containerName="maas-api" containerID="cri-o://7e3f3136c40fcdcb122115e486c1f888b8d5a9864d572d00b69f20e14df0311a" gracePeriod=30 Jul 07 14:42:06.985974 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:06.985942 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:07.039710 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.039679 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:07.180543 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.180521 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" Jul 07 14:42:07.355809 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.355759 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68592e8a-d720-4aa7-97e3-b3d5330cb031-maas-api-tls\") pod \"68592e8a-d720-4aa7-97e3-b3d5330cb031\" (UID: \"68592e8a-d720-4aa7-97e3-b3d5330cb031\") " Jul 07 14:42:07.356020 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.355878 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x682d\" (UniqueName: \"kubernetes.io/projected/68592e8a-d720-4aa7-97e3-b3d5330cb031-kube-api-access-x682d\") pod \"68592e8a-d720-4aa7-97e3-b3d5330cb031\" (UID: \"68592e8a-d720-4aa7-97e3-b3d5330cb031\") " Jul 07 14:42:07.356020 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.355973 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68592e8a-d720-4aa7-97e3-b3d5330cb031-openshift-service-ca\") pod \"68592e8a-d720-4aa7-97e3-b3d5330cb031\" (UID: \"68592e8a-d720-4aa7-97e3-b3d5330cb031\") " Jul 07 14:42:07.356386 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.356352 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68592e8a-d720-4aa7-97e3-b3d5330cb031-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68592e8a-d720-4aa7-97e3-b3d5330cb031" (UID: "68592e8a-d720-4aa7-97e3-b3d5330cb031"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:42:07.358047 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.358023 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68592e8a-d720-4aa7-97e3-b3d5330cb031-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68592e8a-d720-4aa7-97e3-b3d5330cb031" (UID: "68592e8a-d720-4aa7-97e3-b3d5330cb031"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:42:07.358137 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.358085 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68592e8a-d720-4aa7-97e3-b3d5330cb031-kube-api-access-x682d" (OuterVolumeSpecName: "kube-api-access-x682d") pod "68592e8a-d720-4aa7-97e3-b3d5330cb031" (UID: "68592e8a-d720-4aa7-97e3-b3d5330cb031"). InnerVolumeSpecName "kube-api-access-x682d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:42:07.457682 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.457646 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x682d\" (UniqueName: \"kubernetes.io/projected/68592e8a-d720-4aa7-97e3-b3d5330cb031-kube-api-access-x682d\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:42:07.457682 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.457678 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68592e8a-d720-4aa7-97e3-b3d5330cb031-openshift-service-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:42:07.457682 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.457687 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68592e8a-d720-4aa7-97e3-b3d5330cb031-maas-api-tls\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:42:07.553051 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.553016 2560 generic.go:356] "Generic (PLEG): container finished" podID="68592e8a-d720-4aa7-97e3-b3d5330cb031" containerID="7e3f3136c40fcdcb122115e486c1f888b8d5a9864d572d00b69f20e14df0311a" exitCode=0 Jul 07 14:42:07.553222 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.553057 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" Jul 07 14:42:07.553222 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.553108 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" event={"ID":"68592e8a-d720-4aa7-97e3-b3d5330cb031","Type":"ContainerDied","Data":"7e3f3136c40fcdcb122115e486c1f888b8d5a9864d572d00b69f20e14df0311a"} Jul 07 14:42:07.553222 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.553155 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z" event={"ID":"68592e8a-d720-4aa7-97e3-b3d5330cb031","Type":"ContainerDied","Data":"647550a452c8d1f8ca1d2b8c4d75a0f205e7467a52a8fdfb86cb779884702147"} Jul 07 14:42:07.553222 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.553178 2560 scope.go:117] "RemoveContainer" containerID="7e3f3136c40fcdcb122115e486c1f888b8d5a9864d572d00b69f20e14df0311a" Jul 07 14:42:07.564971 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.564951 2560 scope.go:117] "RemoveContainer" containerID="7e3f3136c40fcdcb122115e486c1f888b8d5a9864d572d00b69f20e14df0311a" Jul 07 14:42:07.565253 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:42:07.565234 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e3f3136c40fcdcb122115e486c1f888b8d5a9864d572d00b69f20e14df0311a\": container with ID starting with 7e3f3136c40fcdcb122115e486c1f888b8d5a9864d572d00b69f20e14df0311a not found: ID does not exist" containerID="7e3f3136c40fcdcb122115e486c1f888b8d5a9864d572d00b69f20e14df0311a" Jul 07 14:42:07.565344 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.565259 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e3f3136c40fcdcb122115e486c1f888b8d5a9864d572d00b69f20e14df0311a"} err="failed to get container status \"7e3f3136c40fcdcb122115e486c1f888b8d5a9864d572d00b69f20e14df0311a\": rpc error: code = NotFound desc = could not find container \"7e3f3136c40fcdcb122115e486c1f888b8d5a9864d572d00b69f20e14df0311a\": container with ID starting with 7e3f3136c40fcdcb122115e486c1f888b8d5a9864d572d00b69f20e14df0311a not found: ID does not exist" Jul 07 14:42:07.574234 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.574212 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z"] Jul 07 14:42:07.578555 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:07.578532 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-cdb540-c8d7bc6f6-fkq2z"] Jul 07 14:42:08.209277 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:08.209242 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68592e8a-d720-4aa7-97e3-b3d5330cb031" path="/var/lib/kubelet/pods/68592e8a-d720-4aa7-97e3-b3d5330cb031/volumes" Jul 07 14:42:12.728574 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:12.728538 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:12.827772 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:12.827737 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:12.997745 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:12.997663 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:13.230731 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:13.230696 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:13.391560 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:13.391525 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:13.824325 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:13.824293 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:13.924652 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:13.924621 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:14.035196 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.035162 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn"] Jul 07 14:42:14.035454 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.035424 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" podUID="eeb9e089-eea2-4be4-9892-0a0173733f5a" containerName="maas-api" containerID="cri-o://14a0aee70a69e532f99ae42922199a491fd21a10767955da26eb7466ee4d93c9" gracePeriod=30 Jul 07 14:42:14.093922 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.093839 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:14.286305 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.286258 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" Jul 07 14:42:14.320803 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.320776 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87vgr\" (UniqueName: \"kubernetes.io/projected/eeb9e089-eea2-4be4-9892-0a0173733f5a-kube-api-access-87vgr\") pod \"eeb9e089-eea2-4be4-9892-0a0173733f5a\" (UID: \"eeb9e089-eea2-4be4-9892-0a0173733f5a\") " Jul 07 14:42:14.320970 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.320838 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb9e089-eea2-4be4-9892-0a0173733f5a-openshift-service-ca\") pod \"eeb9e089-eea2-4be4-9892-0a0173733f5a\" (UID: \"eeb9e089-eea2-4be4-9892-0a0173733f5a\") " Jul 07 14:42:14.320970 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.320863 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb9e089-eea2-4be4-9892-0a0173733f5a-maas-api-tls\") pod \"eeb9e089-eea2-4be4-9892-0a0173733f5a\" (UID: \"eeb9e089-eea2-4be4-9892-0a0173733f5a\") " Jul 07 14:42:14.321242 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.321215 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb9e089-eea2-4be4-9892-0a0173733f5a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eeb9e089-eea2-4be4-9892-0a0173733f5a" (UID: "eeb9e089-eea2-4be4-9892-0a0173733f5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:42:14.322947 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.322920 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb9e089-eea2-4be4-9892-0a0173733f5a-kube-api-access-87vgr" (OuterVolumeSpecName: "kube-api-access-87vgr") pod "eeb9e089-eea2-4be4-9892-0a0173733f5a" (UID: "eeb9e089-eea2-4be4-9892-0a0173733f5a"). InnerVolumeSpecName "kube-api-access-87vgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:42:14.323073 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.322958 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb9e089-eea2-4be4-9892-0a0173733f5a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eeb9e089-eea2-4be4-9892-0a0173733f5a" (UID: "eeb9e089-eea2-4be4-9892-0a0173733f5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:42:14.422668 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.422593 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeb9e089-eea2-4be4-9892-0a0173733f5a-maas-api-tls\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:42:14.422668 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.422620 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87vgr\" (UniqueName: \"kubernetes.io/projected/eeb9e089-eea2-4be4-9892-0a0173733f5a-kube-api-access-87vgr\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:42:14.422668 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.422638 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb9e089-eea2-4be4-9892-0a0173733f5a-openshift-service-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:42:14.591903 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.591864 2560 generic.go:356] "Generic (PLEG): container finished" podID="eeb9e089-eea2-4be4-9892-0a0173733f5a" containerID="14a0aee70a69e532f99ae42922199a491fd21a10767955da26eb7466ee4d93c9" exitCode=0 Jul 07 14:42:14.592079 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.591917 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" Jul 07 14:42:14.592079 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.591948 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" event={"ID":"eeb9e089-eea2-4be4-9892-0a0173733f5a","Type":"ContainerDied","Data":"14a0aee70a69e532f99ae42922199a491fd21a10767955da26eb7466ee4d93c9"} Jul 07 14:42:14.592079 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.592006 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn" event={"ID":"eeb9e089-eea2-4be4-9892-0a0173733f5a","Type":"ContainerDied","Data":"0a66dae2c4abbef0cc0ad31474f9a782fbbf559e9663d04b0f69a59c28eb9754"} Jul 07 14:42:14.592079 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.592037 2560 scope.go:117] "RemoveContainer" containerID="14a0aee70a69e532f99ae42922199a491fd21a10767955da26eb7466ee4d93c9" Jul 07 14:42:14.601765 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.601744 2560 scope.go:117] "RemoveContainer" containerID="14a0aee70a69e532f99ae42922199a491fd21a10767955da26eb7466ee4d93c9" Jul 07 14:42:14.602039 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:42:14.602019 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a0aee70a69e532f99ae42922199a491fd21a10767955da26eb7466ee4d93c9\": container with ID starting with 14a0aee70a69e532f99ae42922199a491fd21a10767955da26eb7466ee4d93c9 not found: ID does not exist" containerID="14a0aee70a69e532f99ae42922199a491fd21a10767955da26eb7466ee4d93c9" Jul 07 14:42:14.602110 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.602048 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a0aee70a69e532f99ae42922199a491fd21a10767955da26eb7466ee4d93c9"} err="failed to get container status \"14a0aee70a69e532f99ae42922199a491fd21a10767955da26eb7466ee4d93c9\": rpc error: code = NotFound desc = could not find container \"14a0aee70a69e532f99ae42922199a491fd21a10767955da26eb7466ee4d93c9\": container with ID starting with 14a0aee70a69e532f99ae42922199a491fd21a10767955da26eb7466ee4d93c9 not found: ID does not exist" Jul 07 14:42:14.612619 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.612591 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn"] Jul 07 14:42:14.615918 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:14.615897 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-8c73bb-5cd6fdf87d-z6jbn"] Jul 07 14:42:16.208577 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:16.208539 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeb9e089-eea2-4be4-9892-0a0173733f5a" path="/var/lib/kubelet/pods/eeb9e089-eea2-4be4-9892-0a0173733f5a/volumes" Jul 07 14:42:19.649469 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:19.649438 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:19.751258 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:19.751225 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:19.956740 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:19.956640 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:20.171863 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:20.171829 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:20.528242 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:20.528210 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:20.830137 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:20.830109 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:20.961683 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:20.961652 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:23.978240 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:23.978205 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:24.045102 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.045070 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2"] Jul 07 14:42:24.045460 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.045447 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb9e089-eea2-4be4-9892-0a0173733f5a" containerName="maas-api" Jul 07 14:42:24.045511 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.045463 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb9e089-eea2-4be4-9892-0a0173733f5a" containerName="maas-api" Jul 07 14:42:24.045511 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.045476 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68592e8a-d720-4aa7-97e3-b3d5330cb031" containerName="maas-api" Jul 07 14:42:24.045511 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.045481 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="68592e8a-d720-4aa7-97e3-b3d5330cb031" containerName="maas-api" Jul 07 14:42:24.045603 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.045540 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeb9e089-eea2-4be4-9892-0a0173733f5a" containerName="maas-api" Jul 07 14:42:24.045603 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.045548 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="68592e8a-d720-4aa7-97e3-b3d5330cb031" containerName="maas-api" Jul 07 14:42:24.047595 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.047577 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:42:24.049888 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.049868 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-cc2ffa-serving-cert\"" Jul 07 14:42:24.064294 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.061157 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2"] Jul 07 14:42:24.065733 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.065710 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:24.114579 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.114546 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e11f2f-5e33-47f7-8530-7c46050ebdec-openshift-service-ca\") pod \"maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2\" (UID: \"41e11f2f-5e33-47f7-8530-7c46050ebdec\") " pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:42:24.114579 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.114582 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52jg9\" (UniqueName: \"kubernetes.io/projected/41e11f2f-5e33-47f7-8530-7c46050ebdec-kube-api-access-52jg9\") pod \"maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2\" (UID: \"41e11f2f-5e33-47f7-8530-7c46050ebdec\") " pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:42:24.114779 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.114649 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e11f2f-5e33-47f7-8530-7c46050ebdec-maas-api-tls\") pod \"maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2\" (UID: \"41e11f2f-5e33-47f7-8530-7c46050ebdec\") " pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:42:24.124451 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.124424 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:24.215871 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.215838 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e11f2f-5e33-47f7-8530-7c46050ebdec-openshift-service-ca\") pod \"maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2\" (UID: \"41e11f2f-5e33-47f7-8530-7c46050ebdec\") " pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:42:24.215871 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.215872 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52jg9\" (UniqueName: \"kubernetes.io/projected/41e11f2f-5e33-47f7-8530-7c46050ebdec-kube-api-access-52jg9\") pod \"maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2\" (UID: \"41e11f2f-5e33-47f7-8530-7c46050ebdec\") " pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:42:24.216121 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.215915 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e11f2f-5e33-47f7-8530-7c46050ebdec-maas-api-tls\") pod \"maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2\" (UID: \"41e11f2f-5e33-47f7-8530-7c46050ebdec\") " pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:42:24.216121 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:42:24.216009 2560 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-cc2ffa-serving-cert: secret "maas-api-e2e-shared-a-cc2ffa-serving-cert" not found Jul 07 14:42:24.216121 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:42:24.216067 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41e11f2f-5e33-47f7-8530-7c46050ebdec-maas-api-tls podName:41e11f2f-5e33-47f7-8530-7c46050ebdec nodeName:}" failed. No retries permitted until 2026-07-07 14:42:24.716052522 +0000 UTC m=+2103.148295002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41e11f2f-5e33-47f7-8530-7c46050ebdec-maas-api-tls") pod "maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" (UID: "41e11f2f-5e33-47f7-8530-7c46050ebdec") : secret "maas-api-e2e-shared-a-cc2ffa-serving-cert" not found Jul 07 14:42:24.216607 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.216583 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e11f2f-5e33-47f7-8530-7c46050ebdec-openshift-service-ca\") pod \"maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2\" (UID: \"41e11f2f-5e33-47f7-8530-7c46050ebdec\") " pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:42:24.224061 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.224039 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52jg9\" (UniqueName: \"kubernetes.io/projected/41e11f2f-5e33-47f7-8530-7c46050ebdec-kube-api-access-52jg9\") pod \"maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2\" (UID: \"41e11f2f-5e33-47f7-8530-7c46050ebdec\") " pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:42:24.422443 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.422386 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:24.523093 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.523065 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:24.626242 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.626207 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:24.721187 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.721108 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e11f2f-5e33-47f7-8530-7c46050ebdec-maas-api-tls\") pod \"maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2\" (UID: \"41e11f2f-5e33-47f7-8530-7c46050ebdec\") " pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:42:24.723632 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.723610 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e11f2f-5e33-47f7-8530-7c46050ebdec-maas-api-tls\") pod \"maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2\" (UID: \"41e11f2f-5e33-47f7-8530-7c46050ebdec\") " pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:42:24.962187 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:24.962160 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:42:25.098135 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:25.098110 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2"] Jul 07 14:42:25.100143 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:42:25.100117 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e11f2f_5e33_47f7_8530_7c46050ebdec.slice/crio-fa7575c2a6fddd46c9d35143d80915d4911ea850021d89650298ede1a62032c4 WatchSource:0}: Error finding container fa7575c2a6fddd46c9d35143d80915d4911ea850021d89650298ede1a62032c4: Status 404 returned error can't find the container with id fa7575c2a6fddd46c9d35143d80915d4911ea850021d89650298ede1a62032c4 Jul 07 14:42:25.649687 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:25.649654 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" event={"ID":"41e11f2f-5e33-47f7-8530-7c46050ebdec","Type":"ContainerStarted","Data":"fa7575c2a6fddd46c9d35143d80915d4911ea850021d89650298ede1a62032c4"} Jul 07 14:42:26.657215 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:26.657049 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" event={"ID":"41e11f2f-5e33-47f7-8530-7c46050ebdec","Type":"ContainerStarted","Data":"e7dd2a3b71e249ce7ffb1c97c4447afa17aeb30791d70fcb76346b2542ea4b87"} Jul 07 14:42:26.657860 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:26.657768 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:42:26.676115 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:26.676065 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" podStartSLOduration=2.027408923 podStartE2EDuration="2.67605194s" totalImagesPullingTime="648.643017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:42:26.67495581 +0000 UTC m=+2105.107198334" watchObservedRunningTime="2026-07-07 14:42:26.67605194 +0000 UTC m=+2105.108294441" Jul 07 14:42:28.460878 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:28.460845 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:28.804403 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:28.804371 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:29.127903 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:29.127827 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:29.269407 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:29.269375 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:29.957934 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:29.957905 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:31.820503 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:31.820471 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:31.894366 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:31.894338 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:31.900539 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:31.900510 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2"] Jul 07 14:42:31.903819 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:31.903797 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" Jul 07 14:42:31.906116 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:31.906092 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-332265-serving-cert\"" Jul 07 14:42:31.916853 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:31.916830 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2"] Jul 07 14:42:31.963442 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:31.963410 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:31.990489 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:31.990456 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2d2\" (UniqueName: \"kubernetes.io/projected/7a6df82b-178d-4ed9-afa2-ed076537c2ce-kube-api-access-6h2d2\") pod \"maas-api-e2e-shared-b-332265-6bfffdf689-kscc2\" (UID: \"7a6df82b-178d-4ed9-afa2-ed076537c2ce\") " pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" Jul 07 14:42:31.990593 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:31.990519 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6df82b-178d-4ed9-afa2-ed076537c2ce-maas-api-tls\") pod \"maas-api-e2e-shared-b-332265-6bfffdf689-kscc2\" (UID: \"7a6df82b-178d-4ed9-afa2-ed076537c2ce\") " pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" Jul 07 14:42:31.990645 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:31.990630 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6df82b-178d-4ed9-afa2-ed076537c2ce-openshift-service-ca\") pod \"maas-api-e2e-shared-b-332265-6bfffdf689-kscc2\" (UID: \"7a6df82b-178d-4ed9-afa2-ed076537c2ce\") " pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" Jul 07 14:42:32.091696 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:32.091615 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6df82b-178d-4ed9-afa2-ed076537c2ce-openshift-service-ca\") pod \"maas-api-e2e-shared-b-332265-6bfffdf689-kscc2\" (UID: \"7a6df82b-178d-4ed9-afa2-ed076537c2ce\") " pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" Jul 07 14:42:32.091696 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:32.091672 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2d2\" (UniqueName: \"kubernetes.io/projected/7a6df82b-178d-4ed9-afa2-ed076537c2ce-kube-api-access-6h2d2\") pod \"maas-api-e2e-shared-b-332265-6bfffdf689-kscc2\" (UID: \"7a6df82b-178d-4ed9-afa2-ed076537c2ce\") " pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" Jul 07 14:42:32.091888 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:32.091701 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6df82b-178d-4ed9-afa2-ed076537c2ce-maas-api-tls\") pod \"maas-api-e2e-shared-b-332265-6bfffdf689-kscc2\" (UID: \"7a6df82b-178d-4ed9-afa2-ed076537c2ce\") " pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" Jul 07 14:42:32.092277 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:32.092245 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6df82b-178d-4ed9-afa2-ed076537c2ce-openshift-service-ca\") pod \"maas-api-e2e-shared-b-332265-6bfffdf689-kscc2\" (UID: \"7a6df82b-178d-4ed9-afa2-ed076537c2ce\") " pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" Jul 07 14:42:32.094172 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:32.094149 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6df82b-178d-4ed9-afa2-ed076537c2ce-maas-api-tls\") pod \"maas-api-e2e-shared-b-332265-6bfffdf689-kscc2\" (UID: \"7a6df82b-178d-4ed9-afa2-ed076537c2ce\") " pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" Jul 07 14:42:32.099215 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:32.099195 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2d2\" (UniqueName: \"kubernetes.io/projected/7a6df82b-178d-4ed9-afa2-ed076537c2ce-kube-api-access-6h2d2\") pod \"maas-api-e2e-shared-b-332265-6bfffdf689-kscc2\" (UID: \"7a6df82b-178d-4ed9-afa2-ed076537c2ce\") " pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" Jul 07 14:42:32.109698 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:32.109673 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:32.217036 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:32.217007 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" Jul 07 14:42:32.352626 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:32.352596 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2"] Jul 07 14:42:32.354463 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:42:32.354440 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6df82b_178d_4ed9_afa2_ed076537c2ce.slice/crio-7efc566aa3eb178f2dd880800bbc7b2955247895ed2c68051a6826c656783102 WatchSource:0}: Error finding container 7efc566aa3eb178f2dd880800bbc7b2955247895ed2c68051a6826c656783102: Status 404 returned error can't find the container with id 7efc566aa3eb178f2dd880800bbc7b2955247895ed2c68051a6826c656783102 Jul 07 14:42:32.408045 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:32.408014 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:32.505368 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:32.505335 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:32.667429 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:32.667345 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:42:32.688946 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:32.688915 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" event={"ID":"7a6df82b-178d-4ed9-afa2-ed076537c2ce","Type":"ContainerStarted","Data":"7efc566aa3eb178f2dd880800bbc7b2955247895ed2c68051a6826c656783102"} Jul 07 14:42:33.696869 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:33.696838 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" event={"ID":"7a6df82b-178d-4ed9-afa2-ed076537c2ce","Type":"ContainerStarted","Data":"7d1ae7f1cccb8eff7cfaae235b5e17bb8d20593f85ae3490ec0a521d46db4dc2"} Jul 07 14:42:33.697303 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:33.696938 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" Jul 07 14:42:33.711976 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:33.711913 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" podStartSLOduration=2.256739796 podStartE2EDuration="2.711895636s" totalImagesPullingTime="455.15584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:42:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:42:33.711081894 +0000 UTC m=+2112.143324399" watchObservedRunningTime="2026-07-07 14:42:33.711895636 +0000 UTC m=+2112.144138138" Jul 07 14:42:36.325795 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:36.325763 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:39.708463 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:39.708433 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" Jul 07 14:42:41.977020 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:41.976986 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:42.077951 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:42.077921 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:45.813026 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:45.812996 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:46.270215 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:46.270134 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:46.370673 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:46.370645 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:53.062758 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:53.062710 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:53.163289 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:42:53.163228 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:16.469655 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:43:16.469558 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:43:27.264740 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:43:27.264709 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:32.628633 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:44:32.628593 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:00.135278 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.135146 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29723925-8j8cx"] Jul 07 14:45:00.138460 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.138439 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j"] Jul 07 14:45:00.138598 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.138551 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723925-8j8cx" Jul 07 14:45:00.144925 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.144891 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89"] Jul 07 14:45:00.145216 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.145195 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j" Jul 07 14:45:00.150401 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.149425 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89" Jul 07 14:45:00.152914 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.152891 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89"] Jul 07 14:45:00.156002 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.155983 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723925-8j8cx"] Jul 07 14:45:00.157825 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.157802 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j"] Jul 07 14:45:00.231058 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.231028 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpp9l\" (UniqueName: \"kubernetes.io/projected/ad5a864e-4af3-461f-8bcd-0758b6407e7b-kube-api-access-wpp9l\") pod \"maas-api-key-cleanup-29723925-8j8cx\" (UID: \"ad5a864e-4af3-461f-8bcd-0758b6407e7b\") " pod="opendatahub/maas-api-key-cleanup-29723925-8j8cx" Jul 07 14:45:00.231220 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.231083 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bn29\" (UniqueName: \"kubernetes.io/projected/49524953-a3b2-4891-8759-f9a218537a7d-kube-api-access-6bn29\") pod \"maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89\" (UID: \"49524953-a3b2-4891-8759-f9a218537a7d\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89" Jul 07 14:45:00.231220 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.231160 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9chn\" (UniqueName: \"kubernetes.io/projected/cce4d0e8-e31f-4fd4-a2b6-576292c403f6-kube-api-access-h9chn\") pod \"maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j\" (UID: \"cce4d0e8-e31f-4fd4-a2b6-576292c403f6\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j" Jul 07 14:45:00.331854 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.331820 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpp9l\" (UniqueName: \"kubernetes.io/projected/ad5a864e-4af3-461f-8bcd-0758b6407e7b-kube-api-access-wpp9l\") pod \"maas-api-key-cleanup-29723925-8j8cx\" (UID: \"ad5a864e-4af3-461f-8bcd-0758b6407e7b\") " pod="opendatahub/maas-api-key-cleanup-29723925-8j8cx" Jul 07 14:45:00.332074 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.332050 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bn29\" (UniqueName: \"kubernetes.io/projected/49524953-a3b2-4891-8759-f9a218537a7d-kube-api-access-6bn29\") pod \"maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89\" (UID: \"49524953-a3b2-4891-8759-f9a218537a7d\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89" Jul 07 14:45:00.332177 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.332160 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9chn\" (UniqueName: \"kubernetes.io/projected/cce4d0e8-e31f-4fd4-a2b6-576292c403f6-kube-api-access-h9chn\") pod \"maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j\" (UID: \"cce4d0e8-e31f-4fd4-a2b6-576292c403f6\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j" Jul 07 14:45:00.339522 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.339491 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpp9l\" (UniqueName: \"kubernetes.io/projected/ad5a864e-4af3-461f-8bcd-0758b6407e7b-kube-api-access-wpp9l\") pod \"maas-api-key-cleanup-29723925-8j8cx\" (UID: \"ad5a864e-4af3-461f-8bcd-0758b6407e7b\") " pod="opendatahub/maas-api-key-cleanup-29723925-8j8cx" Jul 07 14:45:00.339639 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.339571 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bn29\" (UniqueName: \"kubernetes.io/projected/49524953-a3b2-4891-8759-f9a218537a7d-kube-api-access-6bn29\") pod \"maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89\" (UID: \"49524953-a3b2-4891-8759-f9a218537a7d\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89" Jul 07 14:45:00.339639 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.339571 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9chn\" (UniqueName: \"kubernetes.io/projected/cce4d0e8-e31f-4fd4-a2b6-576292c403f6-kube-api-access-h9chn\") pod \"maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j\" (UID: \"cce4d0e8-e31f-4fd4-a2b6-576292c403f6\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j" Jul 07 14:45:00.456744 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.456649 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723925-8j8cx" Jul 07 14:45:00.466043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.466019 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j" Jul 07 14:45:00.471155 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.471122 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89" Jul 07 14:45:00.617508 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.617475 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29723925-8j8cx"] Jul 07 14:45:00.618337 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:45:00.618308 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5a864e_4af3_461f_8bcd_0758b6407e7b.slice/crio-42df74c41244462739fa0157befff5d3060a111656355654fb1b22a6be13e996 WatchSource:0}: Error finding container 42df74c41244462739fa0157befff5d3060a111656355654fb1b22a6be13e996: Status 404 returned error can't find the container with id 42df74c41244462739fa0157befff5d3060a111656355654fb1b22a6be13e996 Jul 07 14:45:00.620432 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.620411 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:45:00.852245 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.852217 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89"] Jul 07 14:45:00.854189 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:45:00.854168 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49524953_a3b2_4891_8759_f9a218537a7d.slice/crio-4b95df8f0f106de6836fdb1c0bd0a3c82abdd90e6b60303b04cb8bc57a9c8b70 WatchSource:0}: Error finding container 4b95df8f0f106de6836fdb1c0bd0a3c82abdd90e6b60303b04cb8bc57a9c8b70: Status 404 returned error can't find the container with id 4b95df8f0f106de6836fdb1c0bd0a3c82abdd90e6b60303b04cb8bc57a9c8b70 Jul 07 14:45:00.854609 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:00.854590 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j"] Jul 07 14:45:00.856634 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:45:00.856611 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce4d0e8_e31f_4fd4_a2b6_576292c403f6.slice/crio-2a4867fd43ed8ea2420635e7c122cf5abaf174df152dfa76931113a998bb0120 WatchSource:0}: Error finding container 2a4867fd43ed8ea2420635e7c122cf5abaf174df152dfa76931113a998bb0120: Status 404 returned error can't find the container with id 2a4867fd43ed8ea2420635e7c122cf5abaf174df152dfa76931113a998bb0120 Jul 07 14:45:01.482895 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:01.482846 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723925-8j8cx" event={"ID":"ad5a864e-4af3-461f-8bcd-0758b6407e7b","Type":"ContainerStarted","Data":"42df74c41244462739fa0157befff5d3060a111656355654fb1b22a6be13e996"} Jul 07 14:45:01.484406 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:01.484375 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89" event={"ID":"49524953-a3b2-4891-8759-f9a218537a7d","Type":"ContainerStarted","Data":"4b95df8f0f106de6836fdb1c0bd0a3c82abdd90e6b60303b04cb8bc57a9c8b70"} Jul 07 14:45:01.485876 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:01.485847 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j" event={"ID":"cce4d0e8-e31f-4fd4-a2b6-576292c403f6","Type":"ContainerStarted","Data":"2a4867fd43ed8ea2420635e7c122cf5abaf174df152dfa76931113a998bb0120"} Jul 07 14:45:03.499769 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:03.499677 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89" event={"ID":"49524953-a3b2-4891-8759-f9a218537a7d","Type":"ContainerStarted","Data":"fdd16ee5ae2f038a36adee6684c45f16c4511e9713086ed22aaf3753018dc235"} Jul 07 14:45:03.501239 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:03.501214 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j" event={"ID":"cce4d0e8-e31f-4fd4-a2b6-576292c403f6","Type":"ContainerStarted","Data":"5764c01e7c8e2db8f25a043db1e83305e1efb180d8db98cf0808d65866e8fd54"} Jul 07 14:45:03.502886 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:03.502866 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723925-8j8cx" event={"ID":"ad5a864e-4af3-461f-8bcd-0758b6407e7b","Type":"ContainerStarted","Data":"229f14cf952d56cfe287d5b1d383dec66250d636bd1cf2662495fd82cfc1fe36"} Jul 07 14:45:03.515652 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:03.515607 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89" podStartSLOduration=1.413064154 podStartE2EDuration="3.515590956s" totalImagesPullingTime="2.102526802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:45:03.513131035 +0000 UTC m=+2261.945373534" watchObservedRunningTime="2026-07-07 14:45:03.515590956 +0000 UTC m=+2261.947833457" Jul 07 14:45:03.529379 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:03.529328 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29723925-8j8cx" podStartSLOduration=1.190263414 podStartE2EDuration="3.529308944s" totalImagesPullingTime="2.33904553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:45:03.526383222 +0000 UTC m=+2261.958625723" watchObservedRunningTime="2026-07-07 14:45:03.529308944 +0000 UTC m=+2261.961551447" Jul 07 14:45:03.542247 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:03.542197 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j" podStartSLOduration=1.436824646 podStartE2EDuration="3.542182458s" totalImagesPullingTime="2.105357812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:45:03.541045257 +0000 UTC m=+2261.973287759" watchObservedRunningTime="2026-07-07 14:45:03.542182458 +0000 UTC m=+2261.974424958" Jul 07 14:45:04.509510 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:04.509473 2560 generic.go:356] "Generic (PLEG): container finished" podID="cce4d0e8-e31f-4fd4-a2b6-576292c403f6" containerID="5764c01e7c8e2db8f25a043db1e83305e1efb180d8db98cf0808d65866e8fd54" exitCode=0 Jul 07 14:45:04.509898 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:04.509550 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j" event={"ID":"cce4d0e8-e31f-4fd4-a2b6-576292c403f6","Type":"ContainerDied","Data":"5764c01e7c8e2db8f25a043db1e83305e1efb180d8db98cf0808d65866e8fd54"} Jul 07 14:45:04.511057 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:04.511035 2560 generic.go:356] "Generic (PLEG): container finished" podID="ad5a864e-4af3-461f-8bcd-0758b6407e7b" containerID="229f14cf952d56cfe287d5b1d383dec66250d636bd1cf2662495fd82cfc1fe36" exitCode=0 Jul 07 14:45:04.511170 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:04.511113 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723925-8j8cx" event={"ID":"ad5a864e-4af3-461f-8bcd-0758b6407e7b","Type":"ContainerDied","Data":"229f14cf952d56cfe287d5b1d383dec66250d636bd1cf2662495fd82cfc1fe36"} Jul 07 14:45:04.512611 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:04.512581 2560 generic.go:356] "Generic (PLEG): container finished" podID="49524953-a3b2-4891-8759-f9a218537a7d" containerID="fdd16ee5ae2f038a36adee6684c45f16c4511e9713086ed22aaf3753018dc235" exitCode=0 Jul 07 14:45:04.512716 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:04.512609 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89" event={"ID":"49524953-a3b2-4891-8759-f9a218537a7d","Type":"ContainerDied","Data":"fdd16ee5ae2f038a36adee6684c45f16c4511e9713086ed22aaf3753018dc235"} Jul 07 14:45:05.649754 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:05.649728 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j" Jul 07 14:45:05.704145 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:05.704121 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723925-8j8cx" Jul 07 14:45:05.721228 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:05.721016 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89" Jul 07 14:45:05.787106 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:05.787022 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9chn\" (UniqueName: \"kubernetes.io/projected/cce4d0e8-e31f-4fd4-a2b6-576292c403f6-kube-api-access-h9chn\") pod \"cce4d0e8-e31f-4fd4-a2b6-576292c403f6\" (UID: \"cce4d0e8-e31f-4fd4-a2b6-576292c403f6\") " Jul 07 14:45:05.787106 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:05.787079 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpp9l\" (UniqueName: \"kubernetes.io/projected/ad5a864e-4af3-461f-8bcd-0758b6407e7b-kube-api-access-wpp9l\") pod \"ad5a864e-4af3-461f-8bcd-0758b6407e7b\" (UID: \"ad5a864e-4af3-461f-8bcd-0758b6407e7b\") " Jul 07 14:45:05.789209 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:05.789174 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce4d0e8-e31f-4fd4-a2b6-576292c403f6-kube-api-access-h9chn" (OuterVolumeSpecName: "kube-api-access-h9chn") pod "cce4d0e8-e31f-4fd4-a2b6-576292c403f6" (UID: "cce4d0e8-e31f-4fd4-a2b6-576292c403f6"). InnerVolumeSpecName "kube-api-access-h9chn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:45:05.789209 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:05.789199 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5a864e-4af3-461f-8bcd-0758b6407e7b-kube-api-access-wpp9l" (OuterVolumeSpecName: "kube-api-access-wpp9l") pod "ad5a864e-4af3-461f-8bcd-0758b6407e7b" (UID: "ad5a864e-4af3-461f-8bcd-0758b6407e7b"). InnerVolumeSpecName "kube-api-access-wpp9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:45:05.888656 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:05.888620 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bn29\" (UniqueName: \"kubernetes.io/projected/49524953-a3b2-4891-8759-f9a218537a7d-kube-api-access-6bn29\") pod \"49524953-a3b2-4891-8759-f9a218537a7d\" (UID: \"49524953-a3b2-4891-8759-f9a218537a7d\") " Jul 07 14:45:05.888818 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:05.888773 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9chn\" (UniqueName: \"kubernetes.io/projected/cce4d0e8-e31f-4fd4-a2b6-576292c403f6-kube-api-access-h9chn\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:45:05.888818 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:05.888788 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpp9l\" (UniqueName: \"kubernetes.io/projected/ad5a864e-4af3-461f-8bcd-0758b6407e7b-kube-api-access-wpp9l\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:45:05.890837 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:05.890816 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49524953-a3b2-4891-8759-f9a218537a7d-kube-api-access-6bn29" (OuterVolumeSpecName: "kube-api-access-6bn29") pod "49524953-a3b2-4891-8759-f9a218537a7d" (UID: "49524953-a3b2-4891-8759-f9a218537a7d"). InnerVolumeSpecName "kube-api-access-6bn29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:45:05.990114 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:05.990074 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bn29\" (UniqueName: \"kubernetes.io/projected/49524953-a3b2-4891-8759-f9a218537a7d-kube-api-access-6bn29\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:45:06.524737 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:06.524657 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89" Jul 07 14:45:06.524737 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:06.524705 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89" event={"ID":"49524953-a3b2-4891-8759-f9a218537a7d","Type":"ContainerDied","Data":"4b95df8f0f106de6836fdb1c0bd0a3c82abdd90e6b60303b04cb8bc57a9c8b70"} Jul 07 14:45:06.524955 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:06.524750 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b95df8f0f106de6836fdb1c0bd0a3c82abdd90e6b60303b04cb8bc57a9c8b70" Jul 07 14:45:06.526096 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:06.526078 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j" Jul 07 14:45:06.526096 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:06.526085 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j" event={"ID":"cce4d0e8-e31f-4fd4-a2b6-576292c403f6","Type":"ContainerDied","Data":"2a4867fd43ed8ea2420635e7c122cf5abaf174df152dfa76931113a998bb0120"} Jul 07 14:45:06.526300 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:06.526124 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a4867fd43ed8ea2420635e7c122cf5abaf174df152dfa76931113a998bb0120" Jul 07 14:45:06.527535 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:06.527513 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29723925-8j8cx" event={"ID":"ad5a864e-4af3-461f-8bcd-0758b6407e7b","Type":"ContainerDied","Data":"42df74c41244462739fa0157befff5d3060a111656355654fb1b22a6be13e996"} Jul 07 14:45:06.527643 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:06.527555 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42df74c41244462739fa0157befff5d3060a111656355654fb1b22a6be13e996" Jul 07 14:45:06.527643 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:06.527521 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29723925-8j8cx" Jul 07 14:45:19.829368 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:19.829328 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:26.409634 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:26.409596 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:27.309000 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:27.308964 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:38.039080 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:38.039017 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:45:59.148167 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:45:59.148116 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:43.709335 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:46:43.709299 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:44.537627 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:46:44.537594 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:48.535150 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:46:48.535115 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:10.925839 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:10.925802 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:35.264971 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:35.264936 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:42.649637 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:42.649606 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:42.750194 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:42.750161 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:46.945277 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:46.945236 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:47.045578 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:47.045540 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:51.710761 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:51.710730 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:52.509725 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:52.509674 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:53.573218 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:53.573184 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:53.939244 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:53.939161 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:54.040677 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.040614 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:54.143441 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.143404 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2"] Jul 07 14:47:54.143689 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.143653 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" podUID="41e11f2f-5e33-47f7-8530-7c46050ebdec" containerName="maas-api" containerID="cri-o://e7dd2a3b71e249ce7ffb1c97c4447afa17aeb30791d70fcb76346b2542ea4b87" gracePeriod=30 Jul 07 14:47:54.170615 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.170584 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89"] Jul 07 14:47:54.172872 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.172849 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-cc2ffa-29723925-8xq89"] Jul 07 14:47:54.209844 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.209769 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49524953-a3b2-4891-8759-f9a218537a7d" path="/var/lib/kubelet/pods/49524953-a3b2-4891-8759-f9a218537a7d/volumes" Jul 07 14:47:54.248626 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.248592 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:47:54.393735 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.393707 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:47:54.440920 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.440888 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52jg9\" (UniqueName: \"kubernetes.io/projected/41e11f2f-5e33-47f7-8530-7c46050ebdec-kube-api-access-52jg9\") pod \"41e11f2f-5e33-47f7-8530-7c46050ebdec\" (UID: \"41e11f2f-5e33-47f7-8530-7c46050ebdec\") " Jul 07 14:47:54.441121 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.440927 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e11f2f-5e33-47f7-8530-7c46050ebdec-openshift-service-ca\") pod \"41e11f2f-5e33-47f7-8530-7c46050ebdec\" (UID: \"41e11f2f-5e33-47f7-8530-7c46050ebdec\") " Jul 07 14:47:54.441121 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.441009 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e11f2f-5e33-47f7-8530-7c46050ebdec-maas-api-tls\") pod \"41e11f2f-5e33-47f7-8530-7c46050ebdec\" (UID: \"41e11f2f-5e33-47f7-8530-7c46050ebdec\") " Jul 07 14:47:54.441411 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.441385 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e11f2f-5e33-47f7-8530-7c46050ebdec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "41e11f2f-5e33-47f7-8530-7c46050ebdec" (UID: "41e11f2f-5e33-47f7-8530-7c46050ebdec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:47:54.443216 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.443189 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e11f2f-5e33-47f7-8530-7c46050ebdec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "41e11f2f-5e33-47f7-8530-7c46050ebdec" (UID: "41e11f2f-5e33-47f7-8530-7c46050ebdec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:47:54.443216 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.443198 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e11f2f-5e33-47f7-8530-7c46050ebdec-kube-api-access-52jg9" (OuterVolumeSpecName: "kube-api-access-52jg9") pod "41e11f2f-5e33-47f7-8530-7c46050ebdec" (UID: "41e11f2f-5e33-47f7-8530-7c46050ebdec"). InnerVolumeSpecName "kube-api-access-52jg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:47:54.450407 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.450380 2560 generic.go:356] "Generic (PLEG): container finished" podID="41e11f2f-5e33-47f7-8530-7c46050ebdec" containerID="e7dd2a3b71e249ce7ffb1c97c4447afa17aeb30791d70fcb76346b2542ea4b87" exitCode=0 Jul 07 14:47:54.450507 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.450430 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" Jul 07 14:47:54.450507 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.450469 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" event={"ID":"41e11f2f-5e33-47f7-8530-7c46050ebdec","Type":"ContainerDied","Data":"e7dd2a3b71e249ce7ffb1c97c4447afa17aeb30791d70fcb76346b2542ea4b87"} Jul 07 14:47:54.450593 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.450520 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2" event={"ID":"41e11f2f-5e33-47f7-8530-7c46050ebdec","Type":"ContainerDied","Data":"fa7575c2a6fddd46c9d35143d80915d4911ea850021d89650298ede1a62032c4"} Jul 07 14:47:54.450593 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.450551 2560 scope.go:117] "RemoveContainer" containerID="e7dd2a3b71e249ce7ffb1c97c4447afa17aeb30791d70fcb76346b2542ea4b87" Jul 07 14:47:54.462182 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.462162 2560 scope.go:117] "RemoveContainer" containerID="e7dd2a3b71e249ce7ffb1c97c4447afa17aeb30791d70fcb76346b2542ea4b87" Jul 07 14:47:54.462514 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:47:54.462495 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7dd2a3b71e249ce7ffb1c97c4447afa17aeb30791d70fcb76346b2542ea4b87\": container with ID starting with e7dd2a3b71e249ce7ffb1c97c4447afa17aeb30791d70fcb76346b2542ea4b87 not found: ID does not exist" containerID="e7dd2a3b71e249ce7ffb1c97c4447afa17aeb30791d70fcb76346b2542ea4b87" Jul 07 14:47:54.462570 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.462524 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7dd2a3b71e249ce7ffb1c97c4447afa17aeb30791d70fcb76346b2542ea4b87"} err="failed to get container status \"e7dd2a3b71e249ce7ffb1c97c4447afa17aeb30791d70fcb76346b2542ea4b87\": rpc error: code = NotFound desc = could not find container \"e7dd2a3b71e249ce7ffb1c97c4447afa17aeb30791d70fcb76346b2542ea4b87\": container with ID starting with e7dd2a3b71e249ce7ffb1c97c4447afa17aeb30791d70fcb76346b2542ea4b87 not found: ID does not exist" Jul 07 14:47:54.472174 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.472150 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2"] Jul 07 14:47:54.475174 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.475155 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-cc2ffa-5cfd5776d8-bftq2"] Jul 07 14:47:54.541876 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.541844 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e11f2f-5e33-47f7-8530-7c46050ebdec-maas-api-tls\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:47:54.541876 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.541873 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52jg9\" (UniqueName: \"kubernetes.io/projected/41e11f2f-5e33-47f7-8530-7c46050ebdec-kube-api-access-52jg9\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:47:54.541876 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:54.541883 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e11f2f-5e33-47f7-8530-7c46050ebdec-openshift-service-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:47:56.209350 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:56.209318 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e11f2f-5e33-47f7-8530-7c46050ebdec" path="/var/lib/kubelet/pods/41e11f2f-5e33-47f7-8530-7c46050ebdec/volumes" Jul 07 14:47:59.046531 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:47:59.046487 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:35.990668 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:35.990631 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:36.304689 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:36.304654 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:36.544197 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:36.544140 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:36.848357 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:36.848321 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:37.003833 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.003787 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2"] Jul 07 14:48:37.004250 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.004084 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" podUID="7a6df82b-178d-4ed9-afa2-ed076537c2ce" containerName="maas-api" containerID="cri-o://7d1ae7f1cccb8eff7cfaae235b5e17bb8d20593f85ae3490ec0a521d46db4dc2" gracePeriod=30 Jul 07 14:48:37.033564 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.033538 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:37.034254 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.034234 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j"] Jul 07 14:48:37.037620 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.037599 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-332265-29723925-qwp7j"] Jul 07 14:48:37.136064 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.136038 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:37.252187 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.252157 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" Jul 07 14:48:37.329319 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.329282 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6df82b-178d-4ed9-afa2-ed076537c2ce-maas-api-tls\") pod \"7a6df82b-178d-4ed9-afa2-ed076537c2ce\" (UID: \"7a6df82b-178d-4ed9-afa2-ed076537c2ce\") " Jul 07 14:48:37.329509 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.329359 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h2d2\" (UniqueName: \"kubernetes.io/projected/7a6df82b-178d-4ed9-afa2-ed076537c2ce-kube-api-access-6h2d2\") pod \"7a6df82b-178d-4ed9-afa2-ed076537c2ce\" (UID: \"7a6df82b-178d-4ed9-afa2-ed076537c2ce\") " Jul 07 14:48:37.329509 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.329463 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6df82b-178d-4ed9-afa2-ed076537c2ce-openshift-service-ca\") pod \"7a6df82b-178d-4ed9-afa2-ed076537c2ce\" (UID: \"7a6df82b-178d-4ed9-afa2-ed076537c2ce\") " Jul 07 14:48:37.329875 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.329822 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6df82b-178d-4ed9-afa2-ed076537c2ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a6df82b-178d-4ed9-afa2-ed076537c2ce" (UID: "7a6df82b-178d-4ed9-afa2-ed076537c2ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:48:37.331522 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.331495 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6df82b-178d-4ed9-afa2-ed076537c2ce-kube-api-access-6h2d2" (OuterVolumeSpecName: "kube-api-access-6h2d2") pod "7a6df82b-178d-4ed9-afa2-ed076537c2ce" (UID: "7a6df82b-178d-4ed9-afa2-ed076537c2ce"). InnerVolumeSpecName "kube-api-access-6h2d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:48:37.331657 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.331639 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6df82b-178d-4ed9-afa2-ed076537c2ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a6df82b-178d-4ed9-afa2-ed076537c2ce" (UID: "7a6df82b-178d-4ed9-afa2-ed076537c2ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:48:37.430720 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.430626 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h2d2\" (UniqueName: \"kubernetes.io/projected/7a6df82b-178d-4ed9-afa2-ed076537c2ce-kube-api-access-6h2d2\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:48:37.430720 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.430653 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6df82b-178d-4ed9-afa2-ed076537c2ce-openshift-service-ca\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:48:37.430720 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.430665 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6df82b-178d-4ed9-afa2-ed076537c2ce-maas-api-tls\") on node \"ip-10-0-138-57.ec2.internal\" DevicePath \"\"" Jul 07 14:48:37.691816 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.691715 2560 generic.go:356] "Generic (PLEG): container finished" podID="7a6df82b-178d-4ed9-afa2-ed076537c2ce" containerID="7d1ae7f1cccb8eff7cfaae235b5e17bb8d20593f85ae3490ec0a521d46db4dc2" exitCode=0 Jul 07 14:48:37.691816 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.691768 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" Jul 07 14:48:37.691816 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.691775 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" event={"ID":"7a6df82b-178d-4ed9-afa2-ed076537c2ce","Type":"ContainerDied","Data":"7d1ae7f1cccb8eff7cfaae235b5e17bb8d20593f85ae3490ec0a521d46db4dc2"} Jul 07 14:48:37.692098 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.691829 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2" event={"ID":"7a6df82b-178d-4ed9-afa2-ed076537c2ce","Type":"ContainerDied","Data":"7efc566aa3eb178f2dd880800bbc7b2955247895ed2c68051a6826c656783102"} Jul 07 14:48:37.692098 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.691859 2560 scope.go:117] "RemoveContainer" containerID="7d1ae7f1cccb8eff7cfaae235b5e17bb8d20593f85ae3490ec0a521d46db4dc2" Jul 07 14:48:37.702543 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.702524 2560 scope.go:117] "RemoveContainer" containerID="7d1ae7f1cccb8eff7cfaae235b5e17bb8d20593f85ae3490ec0a521d46db4dc2" Jul 07 14:48:37.702840 ip-10-0-138-57 kubenswrapper[2560]: E0707 14:48:37.702810 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d1ae7f1cccb8eff7cfaae235b5e17bb8d20593f85ae3490ec0a521d46db4dc2\": container with ID starting with 7d1ae7f1cccb8eff7cfaae235b5e17bb8d20593f85ae3490ec0a521d46db4dc2 not found: ID does not exist" containerID="7d1ae7f1cccb8eff7cfaae235b5e17bb8d20593f85ae3490ec0a521d46db4dc2" Jul 07 14:48:37.702908 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.702852 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d1ae7f1cccb8eff7cfaae235b5e17bb8d20593f85ae3490ec0a521d46db4dc2"} err="failed to get container status \"7d1ae7f1cccb8eff7cfaae235b5e17bb8d20593f85ae3490ec0a521d46db4dc2\": rpc error: code = NotFound desc = could not find container \"7d1ae7f1cccb8eff7cfaae235b5e17bb8d20593f85ae3490ec0a521d46db4dc2\": container with ID starting with 7d1ae7f1cccb8eff7cfaae235b5e17bb8d20593f85ae3490ec0a521d46db4dc2 not found: ID does not exist" Jul 07 14:48:37.713419 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.713397 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2"] Jul 07 14:48:37.720053 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:37.720021 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-332265-6bfffdf689-kscc2"] Jul 07 14:48:38.209630 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:38.209594 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a6df82b-178d-4ed9-afa2-ed076537c2ce" path="/var/lib/kubelet/pods/7a6df82b-178d-4ed9-afa2-ed076537c2ce/volumes" Jul 07 14:48:38.210006 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:38.209990 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cce4d0e8-e31f-4fd4-a2b6-576292c403f6" path="/var/lib/kubelet/pods/cce4d0e8-e31f-4fd4-a2b6-576292c403f6/volumes" Jul 07 14:48:42.933878 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:42.933845 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:42.999837 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:42.999795 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:43.191123 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:43.191030 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:43.429459 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:43.429425 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:48.843706 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:48.843673 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-77476c4d87-vq4hq_a8f3a26c-a61e-49b0-9ea9-730ed0c4f483/maas-api/0.log" Jul 07 14:48:49.100746 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:49.100668 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723925-8j8cx_ad5a864e-4af3-461f-8bcd-0758b6407e7b/cleanup/0.log" Jul 07 14:48:49.612534 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:49.612508 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55dcd65d78-5xcq5_6cc32b38-d167-4ef5-acef-19749b6f76a6/copy-manifests/0.log" Jul 07 14:48:49.618692 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:49.618668 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55dcd65d78-5xcq5_6cc32b38-d167-4ef5-acef-19749b6f76a6/manager/0.log" Jul 07 14:48:50.762693 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:50.762663 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj_2a529a61-fb46-46dc-be67-869a613ea05f/util/0.log" Jul 07 14:48:50.768185 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:50.768161 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj_2a529a61-fb46-46dc-be67-869a613ea05f/pull/0.log" Jul 07 14:48:50.773196 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:50.773178 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj_2a529a61-fb46-46dc-be67-869a613ea05f/extract/0.log" Jul 07 14:48:50.878024 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:50.877997 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2_f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669/pull/0.log" Jul 07 14:48:50.883335 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:50.883305 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2_f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669/extract/0.log" Jul 07 14:48:50.888398 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:50.888382 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2_f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669/util/0.log" Jul 07 14:48:51.009873 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:51.009844 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9_9c235c71-4d6c-46ef-a8c5-252a2d6e1480/util/0.log" Jul 07 14:48:51.016407 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:51.016329 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9_9c235c71-4d6c-46ef-a8c5-252a2d6e1480/pull/0.log" Jul 07 14:48:51.022088 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:51.022071 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9_9c235c71-4d6c-46ef-a8c5-252a2d6e1480/extract/0.log" Jul 07 14:48:51.132963 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:51.132926 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf_bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f/util/0.log" Jul 07 14:48:51.138368 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:51.138351 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf_bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f/pull/0.log" Jul 07 14:48:51.161531 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:51.161514 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf_bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f/extract/0.log" Jul 07 14:48:51.847129 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:51.847098 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-wrsbs_7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8/manager/0.log" Jul 07 14:48:52.524456 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:52.524423 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-s69mf_08e49ddb-f2e2-448b-a618-cd3cb935cb7c/discovery/0.log" Jul 07 14:48:53.537408 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:53.537359 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-8jwqp_773a4f6f-bfd0-41e3-830c-884bf82b28dd/main/0.log" Jul 07 14:48:53.990783 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:53.990752 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-zqpfh_8f59b563-8e29-40fa-ba00-4c9e86480a4a/main/0.log" Jul 07 14:48:58.144884 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:58.144847 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:58.202775 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:48:58.202739 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:01.471090 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:01.471063 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xhkww_2b457606-cff0-47a1-8938-778f59dd934d/global-pull-secret-syncer/0.log" Jul 07 14:49:01.517486 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:01.517457 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9wbnm_dc2a0c37-b521-42e0-9e34-bcdc75a54636/konnectivity-agent/0.log" Jul 07 14:49:01.652486 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:01.652460 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-57.ec2.internal_9386b51d35694e977222ff73c9ef8b76/haproxy/0.log" Jul 07 14:49:07.123795 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:07.123767 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj_2a529a61-fb46-46dc-be67-869a613ea05f/extract/0.log" Jul 07 14:49:07.153952 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:07.153923 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj_2a529a61-fb46-46dc-be67-869a613ea05f/util/0.log" Jul 07 14:49:07.174777 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:07.174752 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pl5vj_2a529a61-fb46-46dc-be67-869a613ea05f/pull/0.log" Jul 07 14:49:07.201466 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:07.201432 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2_f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669/extract/0.log" Jul 07 14:49:07.223625 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:07.223603 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2_f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669/util/0.log" Jul 07 14:49:07.246592 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:07.246569 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0dlsw2_f2d8d24f-9fb2-40f7-be6d-aeb41a5f5669/pull/0.log" Jul 07 14:49:07.285765 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:07.285741 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9_9c235c71-4d6c-46ef-a8c5-252a2d6e1480/extract/0.log" Jul 07 14:49:07.329960 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:07.329932 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9_9c235c71-4d6c-46ef-a8c5-252a2d6e1480/util/0.log" Jul 07 14:49:07.383350 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:07.383279 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73pwfp9_9c235c71-4d6c-46ef-a8c5-252a2d6e1480/pull/0.log" Jul 07 14:49:07.428930 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:07.428904 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf_bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f/extract/0.log" Jul 07 14:49:07.451040 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:07.451014 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf_bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f/util/0.log" Jul 07 14:49:07.471398 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:07.471372 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jz5tf_bd6dd2e1-cbbc-44f5-83bb-c1892c2e067f/pull/0.log" Jul 07 14:49:07.964746 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:07.964705 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-wrsbs_7601e6d6-c27c-4522-a0fa-6d3fb8d7c9f8/manager/0.log" Jul 07 14:49:09.742745 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:09.742719 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6bced0b2-a257-435c-ae29-b93f0d8e35e4/alertmanager/0.log" Jul 07 14:49:09.763598 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:09.763573 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6bced0b2-a257-435c-ae29-b93f0d8e35e4/config-reloader/0.log" Jul 07 14:49:09.785329 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:09.785304 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6bced0b2-a257-435c-ae29-b93f0d8e35e4/kube-rbac-proxy-web/0.log" Jul 07 14:49:09.810382 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:09.810360 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6bced0b2-a257-435c-ae29-b93f0d8e35e4/kube-rbac-proxy/0.log" Jul 07 14:49:09.830903 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:09.830870 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6bced0b2-a257-435c-ae29-b93f0d8e35e4/kube-rbac-proxy-metric/0.log" Jul 07 14:49:09.850479 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:09.850451 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6bced0b2-a257-435c-ae29-b93f0d8e35e4/prom-label-proxy/0.log" Jul 07 14:49:09.870512 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:09.870491 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6bced0b2-a257-435c-ae29-b93f0d8e35e4/init-config-reloader/0.log" Jul 07 14:49:09.919515 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:09.919485 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-w76w5_fd5fab7c-5de3-4f92-85a9-863fd04af85d/cluster-monitoring-operator/0.log" Jul 07 14:49:10.022075 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:10.022003 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-f6c6c6887-mgxfb_74877d22-3734-41b4-a37c-158de4eabac0/metrics-server/0.log" Jul 07 14:49:10.045711 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:10.045682 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-zzxlf_f8aec824-073a-42ec-94b8-25c9e535cbdf/monitoring-plugin/0.log" Jul 07 14:49:10.239368 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:10.239341 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nz8cf_de614e33-c9bd-4ba8-850f-8a49fc2b0607/node-exporter/0.log" Jul 07 14:49:10.258656 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:10.258630 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nz8cf_de614e33-c9bd-4ba8-850f-8a49fc2b0607/kube-rbac-proxy/0.log" Jul 07 14:49:10.277534 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:10.277461 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nz8cf_de614e33-c9bd-4ba8-850f-8a49fc2b0607/init-textfile/0.log" Jul 07 14:49:11.833646 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.833543 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h"] Jul 07 14:49:11.834043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.833925 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5a864e-4af3-461f-8bcd-0758b6407e7b" containerName="cleanup" Jul 07 14:49:11.834043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.833939 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5a864e-4af3-461f-8bcd-0758b6407e7b" containerName="cleanup" Jul 07 14:49:11.834043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.833951 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e11f2f-5e33-47f7-8530-7c46050ebdec" containerName="maas-api" Jul 07 14:49:11.834043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.833956 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e11f2f-5e33-47f7-8530-7c46050ebdec" containerName="maas-api" Jul 07 14:49:11.834043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.833968 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49524953-a3b2-4891-8759-f9a218537a7d" containerName="cleanup" Jul 07 14:49:11.834043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.833973 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="49524953-a3b2-4891-8759-f9a218537a7d" containerName="cleanup" Jul 07 14:49:11.834043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.833989 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6df82b-178d-4ed9-afa2-ed076537c2ce" containerName="maas-api" Jul 07 14:49:11.834043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.833994 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6df82b-178d-4ed9-afa2-ed076537c2ce" containerName="maas-api" Jul 07 14:49:11.834043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.834000 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce4d0e8-e31f-4fd4-a2b6-576292c403f6" containerName="cleanup" Jul 07 14:49:11.834043 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.834005 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce4d0e8-e31f-4fd4-a2b6-576292c403f6" containerName="cleanup" Jul 07 14:49:11.834458 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.834064 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cce4d0e8-e31f-4fd4-a2b6-576292c403f6" containerName="cleanup" Jul 07 14:49:11.834458 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.834073 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5a864e-4af3-461f-8bcd-0758b6407e7b" containerName="cleanup" Jul 07 14:49:11.834458 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.834080 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="49524953-a3b2-4891-8759-f9a218537a7d" containerName="cleanup" Jul 07 14:49:11.834458 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.834086 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6df82b-178d-4ed9-afa2-ed076537c2ce" containerName="maas-api" Jul 07 14:49:11.834458 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.834093 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e11f2f-5e33-47f7-8530-7c46050ebdec" containerName="maas-api" Jul 07 14:49:11.837041 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.837023 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:11.839292 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.839251 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tp4qk\"/\"default-dockercfg-rzd2q\"" Jul 07 14:49:11.839394 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.839293 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tp4qk\"/\"kube-root-ca.crt\"" Jul 07 14:49:11.839394 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.839329 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tp4qk\"/\"openshift-service-ca.crt\"" Jul 07 14:49:11.846161 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.846139 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h"] Jul 07 14:49:11.938235 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.938201 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce635c31-d745-42eb-9c11-b2ae557e35fd-sys\") pod \"perf-node-gather-daemonset-bsn4h\" (UID: \"ce635c31-d745-42eb-9c11-b2ae557e35fd\") " pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:11.938235 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.938233 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce635c31-d745-42eb-9c11-b2ae557e35fd-lib-modules\") pod \"perf-node-gather-daemonset-bsn4h\" (UID: \"ce635c31-d745-42eb-9c11-b2ae557e35fd\") " pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:11.938467 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.938249 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce635c31-d745-42eb-9c11-b2ae557e35fd-podres\") pod \"perf-node-gather-daemonset-bsn4h\" (UID: \"ce635c31-d745-42eb-9c11-b2ae557e35fd\") " pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:11.938467 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.938277 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce635c31-d745-42eb-9c11-b2ae557e35fd-proc\") pod \"perf-node-gather-daemonset-bsn4h\" (UID: \"ce635c31-d745-42eb-9c11-b2ae557e35fd\") " pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:11.938467 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:11.938408 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwql7\" (UniqueName: \"kubernetes.io/projected/ce635c31-d745-42eb-9c11-b2ae557e35fd-kube-api-access-cwql7\") pod \"perf-node-gather-daemonset-bsn4h\" (UID: \"ce635c31-d745-42eb-9c11-b2ae557e35fd\") " pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:12.039203 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.039163 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce635c31-d745-42eb-9c11-b2ae557e35fd-sys\") pod \"perf-node-gather-daemonset-bsn4h\" (UID: \"ce635c31-d745-42eb-9c11-b2ae557e35fd\") " pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:12.039203 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.039199 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce635c31-d745-42eb-9c11-b2ae557e35fd-lib-modules\") pod \"perf-node-gather-daemonset-bsn4h\" (UID: \"ce635c31-d745-42eb-9c11-b2ae557e35fd\") " pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:12.039429 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.039219 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce635c31-d745-42eb-9c11-b2ae557e35fd-podres\") pod \"perf-node-gather-daemonset-bsn4h\" (UID: \"ce635c31-d745-42eb-9c11-b2ae557e35fd\") " pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:12.039429 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.039331 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce635c31-d745-42eb-9c11-b2ae557e35fd-sys\") pod \"perf-node-gather-daemonset-bsn4h\" (UID: \"ce635c31-d745-42eb-9c11-b2ae557e35fd\") " pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:12.039429 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.039333 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce635c31-d745-42eb-9c11-b2ae557e35fd-proc\") pod \"perf-node-gather-daemonset-bsn4h\" (UID: \"ce635c31-d745-42eb-9c11-b2ae557e35fd\") " pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:12.039429 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.039374 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce635c31-d745-42eb-9c11-b2ae557e35fd-proc\") pod \"perf-node-gather-daemonset-bsn4h\" (UID: \"ce635c31-d745-42eb-9c11-b2ae557e35fd\") " pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:12.039429 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.039379 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce635c31-d745-42eb-9c11-b2ae557e35fd-lib-modules\") pod \"perf-node-gather-daemonset-bsn4h\" (UID: \"ce635c31-d745-42eb-9c11-b2ae557e35fd\") " pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:12.039429 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.039401 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce635c31-d745-42eb-9c11-b2ae557e35fd-podres\") pod \"perf-node-gather-daemonset-bsn4h\" (UID: \"ce635c31-d745-42eb-9c11-b2ae557e35fd\") " pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:12.039616 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.039492 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwql7\" (UniqueName: \"kubernetes.io/projected/ce635c31-d745-42eb-9c11-b2ae557e35fd-kube-api-access-cwql7\") pod \"perf-node-gather-daemonset-bsn4h\" (UID: \"ce635c31-d745-42eb-9c11-b2ae557e35fd\") " pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:12.046754 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.046729 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwql7\" (UniqueName: \"kubernetes.io/projected/ce635c31-d745-42eb-9c11-b2ae557e35fd-kube-api-access-cwql7\") pod \"perf-node-gather-daemonset-bsn4h\" (UID: \"ce635c31-d745-42eb-9c11-b2ae557e35fd\") " pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:12.149331 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.149235 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:12.285688 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.285663 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h"] Jul 07 14:49:12.287399 ip-10-0-138-57 kubenswrapper[2560]: W0707 14:49:12.287368 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podce635c31_d745_42eb_9c11_b2ae557e35fd.slice/crio-cea14f75482d7dc938b3067a58a4450591f688ffbd5456f1b3bf551f1109f59b WatchSource:0}: Error finding container cea14f75482d7dc938b3067a58a4450591f688ffbd5456f1b3bf551f1109f59b: Status 404 returned error can't find the container with id cea14f75482d7dc938b3067a58a4450591f688ffbd5456f1b3bf551f1109f59b Jul 07 14:49:12.887105 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.887068 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" event={"ID":"ce635c31-d745-42eb-9c11-b2ae557e35fd","Type":"ContainerStarted","Data":"669f510992c1d1700bdcb0b7bb0aa57a92107f02836360d3e0ee65d82a23346c"} Jul 07 14:49:12.887510 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.887137 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:12.887510 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.887158 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" event={"ID":"ce635c31-d745-42eb-9c11-b2ae557e35fd","Type":"ContainerStarted","Data":"cea14f75482d7dc938b3067a58a4450591f688ffbd5456f1b3bf551f1109f59b"} Jul 07 14:49:12.902679 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:12.902626 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" podStartSLOduration=1.902610226 podStartE2EDuration="1.902610226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:49:12.900476279 +0000 UTC m=+2511.332718784" watchObservedRunningTime="2026-07-07 14:49:12.902610226 +0000 UTC m=+2511.334852727" Jul 07 14:49:13.163051 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:13.162973 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7b94988699-mrptp_a27c13ef-4536-428d-b28d-287cf1bddbc0/console/0.log" Jul 07 14:49:13.198436 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:13.198413 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-s7c7g_93bad8d3-1a35-486c-b1e6-539ced4e3b45/download-server/0.log" Jul 07 14:49:14.826577 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:14.826549 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hkczl_b88eafd6-43c8-460b-b450-7b2714fb1a9c/dns/0.log" Jul 07 14:49:14.846576 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:14.846555 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hkczl_b88eafd6-43c8-460b-b450-7b2714fb1a9c/kube-rbac-proxy/0.log" Jul 07 14:49:14.892102 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:14.892075 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6bc6n_1ac773f2-d330-4140-97aa-8ac19ee4d555/dns-node-resolver/0.log" Jul 07 14:49:15.456180 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:15.456149 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d85bcc555-qb8tv_ff734b15-88fd-402d-8c78-0a60cb48c5d8/registry/0.log" Jul 07 14:49:15.465209 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:15.465186 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d85bcc555-qb8tv_ff734b15-88fd-402d-8c78-0a60cb48c5d8/registry/1.log" Jul 07 14:49:15.483549 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:15.483519 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5kt5s_1053d78f-a5ae-4acd-9a99-32beb279bbb4/node-ca/0.log" Jul 07 14:49:16.545785 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:16.545753 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-s69mf_08e49ddb-f2e2-448b-a618-cd3cb935cb7c/discovery/0.log" Jul 07 14:49:17.496344 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:17.496305 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5cgb6_a334bf52-3bf3-4862-8680-d4e89f3d28bc/serve-healthcheck-canary/0.log" Jul 07 14:49:18.021113 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:18.021082 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-hwj87_b9a6eef4-26d4-4d5c-a1ea-6d71968d1116/insights-operator/0.log" Jul 07 14:49:18.022835 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:18.022818 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-hwj87_b9a6eef4-26d4-4d5c-a1ea-6d71968d1116/insights-operator/1.log" Jul 07 14:49:18.123711 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:18.123682 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7xj79_4961cb11-50bf-4289-8a88-4100c26959d6/kube-rbac-proxy/0.log" Jul 07 14:49:18.147892 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:18.147869 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7xj79_4961cb11-50bf-4289-8a88-4100c26959d6/exporter/0.log" Jul 07 14:49:18.169781 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:18.169753 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7xj79_4961cb11-50bf-4289-8a88-4100c26959d6/extractor/0.log" Jul 07 14:49:18.903338 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:18.903311 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tp4qk/perf-node-gather-daemonset-bsn4h" Jul 07 14:49:20.510254 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:20.510222 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-77476c4d87-vq4hq_a8f3a26c-a61e-49b0-9ea9-730ed0c4f483/maas-api/0.log" Jul 07 14:49:20.560563 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:20.560535 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29723925-8j8cx_ad5a864e-4af3-461f-8bcd-0758b6407e7b/cleanup/0.log" Jul 07 14:49:20.823598 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:20.823574 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55dcd65d78-5xcq5_6cc32b38-d167-4ef5-acef-19749b6f76a6/manager/0.log" Jul 07 14:49:20.844787 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:20.844761 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55dcd65d78-5xcq5_6cc32b38-d167-4ef5-acef-19749b6f76a6/copy-manifests/0.log" Jul 07 14:49:22.588512 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:22.588482 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-s69mf_08e49ddb-f2e2-448b-a618-cd3cb935cb7c/discovery/0.log" Jul 07 14:49:22.591853 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:22.591826 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6489488d88-4dbdt_f446dfd7-14e6-46c4-929b-adc563464d1d/manager/0.log" Jul 07 14:49:24.096753 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:24.096717 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-s69mf_08e49ddb-f2e2-448b-a618-cd3cb935cb7c/discovery/0.log" Jul 07 14:49:27.433926 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:27.433897 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-phxnw_aed7476c-67d4-4194-ba84-fa5b9909a0ce/migrator/0.log" Jul 07 14:49:27.452111 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:27.452075 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-phxnw_aed7476c-67d4-4194-ba84-fa5b9909a0ce/graceful-termination/0.log" Jul 07 14:49:29.144032 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:29.144006 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpdnk_6cf123ea-2a30-4535-971f-0833cafaedea/kube-multus-additional-cni-plugins/0.log" Jul 07 14:49:29.163322 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:29.163293 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpdnk_6cf123ea-2a30-4535-971f-0833cafaedea/egress-router-binary-copy/0.log" Jul 07 14:49:29.181836 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:29.181802 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpdnk_6cf123ea-2a30-4535-971f-0833cafaedea/cni-plugins/0.log" Jul 07 14:49:29.202358 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:29.202332 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpdnk_6cf123ea-2a30-4535-971f-0833cafaedea/bond-cni-plugin/0.log" Jul 07 14:49:29.219768 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:29.219742 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpdnk_6cf123ea-2a30-4535-971f-0833cafaedea/routeoverride-cni/0.log" Jul 07 14:49:29.239319 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:29.239293 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpdnk_6cf123ea-2a30-4535-971f-0833cafaedea/whereabouts-cni-bincopy/0.log" Jul 07 14:49:29.257138 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:29.257116 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bpdnk_6cf123ea-2a30-4535-971f-0833cafaedea/whereabouts-cni/0.log" Jul 07 14:49:29.290589 ip-10-0-138-57 kubenswrapper[2560]: I0707 14:49:29.290561 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vqk55_79313367-388a-4b7a-9a7c-88cf9aeb1d2a/kube-multus/0.log"