Jul 14 18:21:36.784958 ip-10-0-132-224 systemd[1]: Starting Kubernetes Kubelet... Jul 14 18:21:37.115390 ip-10-0-132-224 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:21:37.115390 ip-10-0-132-224 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 18:21:37.115390 ip-10-0-132-224 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:21:37.115390 ip-10-0-132-224 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 18:21:37.115390 ip-10-0-132-224 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:21:37.120448 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.116660 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122050 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122061 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122065 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122068 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122071 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122074 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122077 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122080 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122082 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122085 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122088 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122091 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122093 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122099 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122103 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122113 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122117 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122119 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122122 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:37.125960 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122125 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122127 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122131 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122135 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122138 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122141 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122144 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122147 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122149 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122152 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122154 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122157 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122159 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122162 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122165 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122168 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122171 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122173 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122176 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:37.126985 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122179 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122181 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122184 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122186 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122189 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122192 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122194 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122197 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122200 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122203 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122206 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122209 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122211 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122214 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122216 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122219 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122221 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122224 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122226 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122229 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:37.132111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122231 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122234 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122236 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122239 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122242 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122245 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122247 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122250 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122253 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122256 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122259 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122261 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122265 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122267 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122270 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122272 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122275 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122277 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122281 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122283 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:37.236432 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122286 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:37.188089 ip-10-0-132-224 systemd[1]: Started Kubernetes Kubelet. Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122288 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122291 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122294 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122297 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122299 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122302 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122304 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122307 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122309 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122312 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122314 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122317 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122320 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122322 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122325 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122329 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122333 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122336 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122339 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:37.237896 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.122342 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124219 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124233 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124236 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124240 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124243 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124246 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124249 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124254 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124257 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124260 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124264 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124267 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124270 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124272 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124275 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124278 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124280 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124283 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124285 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:37.241187 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124288 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124291 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124294 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124298 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124302 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124310 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124313 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124316 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124318 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124322 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124325 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124328 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124331 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124333 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124337 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124342 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124348 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124350 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124354 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124357 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:37.242264 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124360 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124362 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124365 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124368 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124370 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124373 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124376 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124378 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124381 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124384 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124386 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124389 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124392 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124394 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124397 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124400 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124403 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124406 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124408 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:37.246006 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124413 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124416 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124419 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124422 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124425 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124428 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124431 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124434 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124437 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124440 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124442 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124445 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124447 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124450 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124452 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124455 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124459 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124462 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124464 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124467 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:37.247044 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124470 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124472 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124477 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124479 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124482 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124485 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124488 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124491 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124493 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124496 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124498 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124501 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124504 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124506 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124510 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124514 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124517 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124520 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124523 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124525 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:37.250785 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124528 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124967 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124974 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124977 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124980 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124984 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124987 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124989 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124993 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124995 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.124998 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125001 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125003 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125006 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125011 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125013 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125018 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125023 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125026 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125029 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:38.931445 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125032 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125034 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125037 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125040 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125043 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125046 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125048 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125051 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125053 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125056 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125059 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125062 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125064 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125067 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125069 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125072 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125074 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125077 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125079 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125081 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:38.941033 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125084 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125086 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125089 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125092 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125094 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125096 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125102 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125104 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125107 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125110 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125113 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125116 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125118 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125121 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125123 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125126 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125128 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125131 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125133 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125136 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:38.942373 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125138 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125143 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125146 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125148 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125151 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125154 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125157 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125159 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125162 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125165 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125167 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125170 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125172 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125175 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125177 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125180 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125182 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125185 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125189 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125192 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:38.943206 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125196 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125200 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125202 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125205 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125208 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125210 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125213 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125216 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125218 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125221 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125224 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125226 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125230 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125234 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125237 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125240 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125243 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125246 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125248 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:38.944946 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125251 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125319 2573 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125326 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125334 2573 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125340 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125346 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125349 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125353 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125357 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125361 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125364 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125369 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125372 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125376 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125379 2573 flags.go:64] FLAG: --cgroup-root="" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125381 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125384 2573 flags.go:64] FLAG: --client-ca-file="" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125388 2573 flags.go:64] FLAG: --cloud-provider="external" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125391 2573 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125395 2573 flags.go:64] FLAG: --cluster-domain="" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125397 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125401 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125403 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125407 2573 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 18:21:38.945816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125415 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125418 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125422 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125425 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125428 2573 flags.go:64] FLAG: --contention-profiling="false" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125431 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125434 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125437 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125440 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125444 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125447 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125450 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125452 2573 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125455 2573 flags.go:64] FLAG: --enable-server="true" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125458 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125463 2573 flags.go:64] FLAG: --event-burst="100" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125466 2573 flags.go:64] FLAG: --event-qps="50" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125469 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125472 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125476 2573 flags.go:64] FLAG: --eviction-hard="" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125480 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125483 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125486 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125489 2573 flags.go:64] FLAG: --eviction-soft="" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125492 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 18:21:38.947126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125495 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125498 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125501 2573 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125504 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125507 2573 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125510 2573 flags.go:64] FLAG: --feature-gates="" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125513 2573 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125516 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125519 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125523 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125526 2573 flags.go:64] FLAG: --healthz-port="10248" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125529 2573 flags.go:64] FLAG: --help="false" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125532 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-132-224.ec2.internal" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125535 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125538 2573 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125540 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125543 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125547 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125549 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125552 2573 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125555 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125557 2573 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125560 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125563 2573 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 18:21:38.948009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125566 2573 flags.go:64] FLAG: --kube-reserved="" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125569 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125572 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125575 2573 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125578 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125581 2573 flags.go:64] FLAG: --lock-file="" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125584 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125586 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125589 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125595 2573 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125598 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125601 2573 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125603 2573 flags.go:64] FLAG: --logging-format="text" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125606 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125609 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125612 2573 flags.go:64] FLAG: --manifest-url="" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125615 2573 flags.go:64] FLAG: --manifest-url-header="" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125620 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125623 2573 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125627 2573 flags.go:64] FLAG: --max-pods="110" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125630 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125632 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125636 2573 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125639 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125642 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 18:21:38.949341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125645 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125648 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125655 2573 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125657 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125660 2573 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125663 2573 flags.go:64] FLAG: --pod-cidr="" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125666 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125672 2573 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125675 2573 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125692 2573 flags.go:64] FLAG: --pods-per-core="0" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125698 2573 flags.go:64] FLAG: --port="10250" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125701 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125703 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-021127a9d58cea3a2" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125706 2573 flags.go:64] FLAG: --qos-reserved="" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125710 2573 flags.go:64] FLAG: --read-only-port="10255" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125713 2573 flags.go:64] FLAG: --register-node="true" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125715 2573 flags.go:64] FLAG: --register-with-taints="" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125719 2573 flags.go:64] FLAG: --registry-burst="10" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125722 2573 flags.go:64] FLAG: --registry-qps="5" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125725 2573 flags.go:64] FLAG: --reserved-cpus="" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125728 2573 flags.go:64] FLAG: --reserved-memory="" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125731 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125734 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125737 2573 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125741 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 18:21:38.951574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125744 2573 flags.go:64] FLAG: --runonce="false" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125747 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125749 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125752 2573 flags.go:64] FLAG: --seccomp-default="false" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125755 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125758 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125761 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125764 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125767 2573 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125770 2573 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125773 2573 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125775 2573 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125778 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125781 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125784 2573 flags.go:64] FLAG: --system-cgroups="" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125786 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125791 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125795 2573 flags.go:64] FLAG: --tls-cert-file="" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125798 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125801 2573 flags.go:64] FLAG: --tls-min-version="" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125804 2573 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125807 2573 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125810 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125812 2573 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125815 2573 flags.go:64] FLAG: --v="2" Jul 14 18:21:38.952206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125819 2573 flags.go:64] FLAG: --version="false" Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125823 2573 flags.go:64] FLAG: --vmodule="" Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125827 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.125830 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125942 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125946 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125951 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125954 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125957 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125960 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125963 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125966 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125969 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125972 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125975 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125977 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125980 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125983 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125985 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125988 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:38.952820 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125990 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125993 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125995 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.125998 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126002 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126004 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126007 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126009 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126012 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126014 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126017 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126020 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126022 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126025 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126028 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126031 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126034 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126036 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126040 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126042 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:38.953307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126045 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126048 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126051 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126053 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126055 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126058 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126060 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126063 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126066 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126068 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126071 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126074 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126077 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126081 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126084 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126086 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126090 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126093 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126095 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:38.953840 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126098 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126101 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126105 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126108 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126111 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126114 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126117 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126120 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126123 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126126 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126129 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126133 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126136 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126139 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126142 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126145 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126147 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126150 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126153 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:38.954312 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126155 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126158 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126161 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126163 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126165 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126168 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126170 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126173 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126175 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126178 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126182 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126185 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126187 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126189 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126192 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126194 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126197 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126199 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126202 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:38.954770 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126204 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:38.955223 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126207 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:38.955223 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126209 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:38.955223 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126212 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:38.955223 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126215 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:38.955223 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.126218 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:38.955223 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.126741 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:21:38.955223 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.133336 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 18:21:38.955223 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.133350 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 18:21:38.955223 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.133369 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 18:21:38.955223 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.133375 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 18:21:38.955223 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133428 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:38.955223 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133434 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:38.955223 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133438 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:38.955223 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133441 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133444 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133447 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133450 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133453 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133457 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133460 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133462 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133465 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133467 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133470 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133473 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133476 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133479 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133482 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133484 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133487 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133490 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133493 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133495 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:38.955587 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133498 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133501 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133503 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133506 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133509 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133511 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133514 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133517 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133520 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133522 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133525 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133527 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133530 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133532 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133534 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133537 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133539 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133542 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133544 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133547 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:38.956054 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133549 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133551 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133554 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133557 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133560 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133562 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133565 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133567 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133573 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133576 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133579 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133581 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133584 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133586 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133589 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133592 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133594 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133597 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133600 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133603 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:38.956542 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133606 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133608 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133611 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133615 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133618 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133621 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133624 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133627 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133629 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133632 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133636 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133639 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133643 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133645 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133648 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133651 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133654 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133656 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:38.957016 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133659 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133661 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133665 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133667 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133670 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133673 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133675 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133693 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133697 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133699 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133702 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133705 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133708 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133710 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133717 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133719 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133722 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:38.957454 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133725 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:38.957851 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.133730 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:21:38.957851 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133838 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:38.957851 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133843 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:38.957851 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133846 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:38.957851 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133849 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:38.957851 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133852 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:38.957851 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133854 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:38.957851 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133857 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:38.957851 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133859 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:38.957851 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133862 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:38.957851 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133865 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:38.957851 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133867 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:38.957851 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133870 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:38.957851 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133874 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:38.957851 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133879 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133882 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133884 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133889 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133891 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133894 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133897 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133899 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133902 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133904 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133907 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133909 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133912 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133914 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133917 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133920 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133923 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133925 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133928 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133930 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:38.958205 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133932 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133935 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133938 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133940 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133942 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133945 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133947 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133950 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133952 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133956 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133958 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133961 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133963 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133966 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133969 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133972 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133975 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133978 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133980 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133983 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:38.958696 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133985 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133988 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133991 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133994 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133996 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.133998 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134002 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134006 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134008 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134011 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134014 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134016 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134019 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134021 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134024 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134026 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134029 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134031 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134034 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:38.959181 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134036 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134038 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134042 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134045 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134048 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134050 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134053 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134055 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134061 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134064 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134066 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134069 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134072 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134075 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134077 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134080 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134083 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134085 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134087 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:38.959646 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134090 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:38.960100 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134092 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:38.960100 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134095 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:38.960100 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134098 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:38.960100 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134100 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:38.960100 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134103 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:38.960100 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134105 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:38.960100 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:37.134107 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:38.960100 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.134112 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:21:38.960100 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.134212 2573 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 18:21:38.960100 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.136778 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 18:21:38.960100 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.137987 2573 server.go:1019] "Starting client certificate rotation" Jul 14 18:21:38.960100 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.138080 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 18:21:38.960100 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.138113 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 18:21:38.960439 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.140220 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 18:21:38.960439 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.144704 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 18:21:38.960439 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.156309 2573 log.go:25] "Validated CRI v1 runtime API" Jul 14 18:21:38.960439 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.158338 2573 log.go:25] "Validated CRI v1 image API" Jul 14 18:21:38.960439 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.159078 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 18:21:38.960439 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.161848 2573 fs.go:135] Filesystem UUIDs: map[2bcc7dd1-905a-4992-9809-a2584eccc0b8:/dev/nvme0n1p4 52925830-35ab-428f-ac46-56e64a5f1be1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 14 18:21:38.960439 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.161862 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.166998 2573 manager.go:216] Machine: {Timestamp:2026-07-14 18:21:37.165953699 +0000 UTC m=+0.283004936 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099923 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec288735d2c51ba9173491de76221276 SystemUUID:ec288735-d2c5-1ba9-1734-91de76221276 BootID:a73ef6a1-3f86-4e30-9655-4926f8f91442 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3f:9b:85:7e:a9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3f:9b:85:7e:a9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:b2:a1:a4:8b:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.167100 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.167180 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.169871 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.169900 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-224.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.170537 2573 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.170547 2573 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.170554 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.170660 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.171128 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.172388 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.176226 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.176348 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.176377 2573 kubelet.go:486] "Attempting to sync node with API server" Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.176394 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.176404 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 18:21:38.960662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.176415 2573 kubelet.go:398] "Adding apiserver pod source" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.176424 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.176774 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.176795 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.178624 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.180656 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.180710 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.180914 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.180931 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.180940 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.180948 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.180959 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.180967 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.181432 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.181441 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.181451 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.181460 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.181472 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.181484 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.182732 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.182743 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.186592 2573 server.go:1268] "Started kubelet" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.186653 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.186789 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.186953 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.188197 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.189565 2573 server.go:310] "Adding debug handlers to kubelet server" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:37.192517 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-224.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.192607 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-224.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:37.193007 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:37.193581 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.196749 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.196757 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:37.196880 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-224.ec2.internal\" not found" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.197500 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.197517 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.197649 2573 reconstruct.go:97] "Volume reconstruction finished" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.197659 2573 reconciler.go:29] "Reconciler: start to sync state" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.197666 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.198113 2573 factory.go:55] Registering systemd factory Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.198129 2573 factory.go:223] Registration of the systemd container factory successfully Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.198339 2573 factory.go:153] Registering CRI-O factory Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.198349 2573 factory.go:223] Registration of the crio container factory successfully Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.198401 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.198425 2573 factory.go:103] Registering Raw factory Jul 14 18:21:38.961099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.198439 2573 manager.go:1196] Started watching for new ooms in manager Jul 14 18:21:38.962198 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:37.197819 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-224.ec2.internal.18c23b057f2b4da1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-224.ec2.internal,UID:ip-10-0-132-224.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-224.ec2.internal,},FirstTimestamp:2026-07-14 18:21:37.186557345 +0000 UTC m=+0.303608585,LastTimestamp:2026-07-14 18:21:37.186557345 +0000 UTC m=+0.303608585,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-224.ec2.internal,}" Jul 14 18:21:38.962198 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.199094 2573 manager.go:349] Starting recovery of all containers Jul 14 18:21:38.962198 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.209421 2573 manager.go:354] Recovery completed Jul 14 18:21:38.962198 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:37.213202 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:38.962198 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:37.297370 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-224.ec2.internal\" not found" Jul 14 18:21:38.977338 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:38.977298 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-224.ec2.internal\" not found" Jul 14 18:21:38.998639 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:38.998621 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bmm6j" Jul 14 18:21:39.010834 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.010818 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bmm6j" Jul 14 18:21:39.052957 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.052936 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:39.053057 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.052969 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:39.053057 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.052980 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:39.053973 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.053958 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 18:21:39.053973 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.053972 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 18:21:39.054118 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.053982 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:39.054118 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.053991 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 14 18:21:39.054771 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.054753 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:39.054854 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.054785 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:39.054854 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.054798 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:39.056088 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.056072 2573 policy_none.go:49] "None policy: Start" Jul 14 18:21:39.056150 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.056092 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 18:21:39.056150 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.056105 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 18:21:39.058185 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.058171 2573 policy_none.go:47] "Start" Jul 14 18:21:39.062722 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.062591 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:39.067679 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.067622 2573 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-224.ec2.internal\" not found" node="ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.077464 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.077444 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-224.ec2.internal\" not found" Jul 14 18:21:39.083964 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.083952 2573 manager.go:337] "Starting Device Plugin manager" Jul 14 18:21:39.084030 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.083980 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 18:21:39.084030 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.083989 2573 server.go:87] "Starting device plugin registration server" Jul 14 18:21:39.084210 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.084199 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 18:21:39.084243 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.084212 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 18:21:39.084349 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.084335 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 18:21:39.084468 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.084436 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 18:21:39.084468 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.084448 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 18:21:39.084899 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.084882 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 18:21:39.084962 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.084931 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-224.ec2.internal\" not found" Jul 14 18:21:39.184731 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.184709 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:39.186741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.185642 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:39.186741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.185679 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:39.186741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.185693 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:39.186741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.185717 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.196617 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.196601 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.196697 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.196622 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-224.ec2.internal\": node \"ip-10-0-132-224.ec2.internal\" not found" Jul 14 18:21:39.213849 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.213818 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 18:21:39.215026 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.215011 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 18:21:39.215077 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.215047 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 18:21:39.215077 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.215070 2573 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 18:21:39.215164 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.215111 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 18:21:39.220651 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.220635 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:39.250327 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.250305 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-224.ec2.internal\" not found" Jul 14 18:21:39.315777 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.315744 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-224.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal"] Jul 14 18:21:39.315868 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.315830 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:39.317552 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.317538 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:39.317635 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.317570 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:39.317635 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.317586 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:39.319860 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.319814 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:39.319949 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.319935 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.319993 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.319961 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:39.320558 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.320543 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:39.320628 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.320572 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:39.320628 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.320543 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:39.320628 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.320583 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:39.320628 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.320601 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:39.320628 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.320616 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:39.322813 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.322792 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.322866 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.322826 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:39.323486 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.323472 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:39.323562 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.323493 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:39.323562 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.323506 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:39.346865 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.346845 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-224.ec2.internal\" not found" node="ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.351092 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.351074 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-224.ec2.internal\" not found" Jul 14 18:21:39.351193 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.351179 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-224.ec2.internal\" not found" node="ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.378751 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.378731 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/13b1f4d964eb8d8619f16afcc6da67af-config\") pod \"kube-apiserver-proxy-ip-10-0-132-224.ec2.internal\" (UID: \"13b1f4d964eb8d8619f16afcc6da67af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.378818 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.378754 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8f99cf49d21fb0224d8e1f094bc56ee0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal\" (UID: \"8f99cf49d21fb0224d8e1f094bc56ee0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.378818 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.378771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f99cf49d21fb0224d8e1f094bc56ee0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal\" (UID: \"8f99cf49d21fb0224d8e1f094bc56ee0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.451575 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.451552 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-224.ec2.internal\" not found" Jul 14 18:21:39.478899 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.478882 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/13b1f4d964eb8d8619f16afcc6da67af-config\") pod \"kube-apiserver-proxy-ip-10-0-132-224.ec2.internal\" (UID: \"13b1f4d964eb8d8619f16afcc6da67af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.478960 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.478905 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8f99cf49d21fb0224d8e1f094bc56ee0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal\" (UID: \"8f99cf49d21fb0224d8e1f094bc56ee0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.478960 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.478922 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f99cf49d21fb0224d8e1f094bc56ee0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal\" (UID: \"8f99cf49d21fb0224d8e1f094bc56ee0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.479057 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.478960 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f99cf49d21fb0224d8e1f094bc56ee0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal\" (UID: \"8f99cf49d21fb0224d8e1f094bc56ee0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.479057 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.478978 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/13b1f4d964eb8d8619f16afcc6da67af-config\") pod \"kube-apiserver-proxy-ip-10-0-132-224.ec2.internal\" (UID: \"13b1f4d964eb8d8619f16afcc6da67af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.479057 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.479005 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8f99cf49d21fb0224d8e1f094bc56ee0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal\" (UID: \"8f99cf49d21fb0224d8e1f094bc56ee0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.552043 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.552005 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-224.ec2.internal\" not found" Jul 14 18:21:39.648549 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.648485 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.652159 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.652142 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-224.ec2.internal\" not found" Jul 14 18:21:39.654396 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.654380 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal" Jul 14 18:21:39.717484 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.717460 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 18:21:39.717915 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.717583 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 18:21:39.717915 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:39.717609 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 18:21:39.752732 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.752712 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-224.ec2.internal\" not found" Jul 14 18:21:39.853019 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.852991 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-224.ec2.internal\" not found" Jul 14 18:21:39.953600 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:39.953512 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-224.ec2.internal\" not found" Jul 14 18:21:40.015064 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.015029 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 18:16:39 +0000 UTC" deadline="2028-01-27 14:14:30.524676367 +0000 UTC" Jul 14 18:21:40.015064 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.015060 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13483h52m50.509619897s" Jul 14 18:21:40.044941 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.044919 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:40.077558 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.077527 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-224.ec2.internal" Jul 14 18:21:40.088745 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.088728 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 18:21:40.090200 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.090186 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal" Jul 14 18:21:40.108418 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.108397 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 18:21:40.243798 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.243730 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:40.535842 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.535768 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:40.757895 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.757867 2573 apiserver.go:52] "Watching apiserver" Jul 14 18:21:40.766882 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.766865 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 18:21:40.768835 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.767974 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-24f4c","openshift-multus/multus-additional-cni-plugins-wpz5d","kube-system/konnectivity-agent-66x25","kube-system/kube-apiserver-proxy-ip-10-0-132-224.ec2.internal","openshift-cluster-node-tuning-operator/tuned-7m6w6","openshift-image-registry/node-ca-kjlrj","openshift-multus/network-metrics-daemon-29dsq","openshift-network-diagnostics/network-check-target-rdlqb","openshift-network-operator/iptables-alerter-h59zr","openshift-ovn-kubernetes/ovnkube-node-8pxlk","kube-system/global-pull-secret-syncer-gm8z4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss","openshift-dns/node-resolver-tjpz8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal"] Jul 14 18:21:40.770852 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.770837 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.772908 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.772893 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tjpz8" Jul 14 18:21:40.775050 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.775036 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-66x25" Jul 14 18:21:40.776850 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.776836 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 18:21:40.777194 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.777179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:40.777318 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:40.777278 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:21:40.778187 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.778171 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 18:21:40.778316 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.778304 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hnlpt\"" Jul 14 18:21:40.778405 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.778389 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 18:21:40.778773 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.778753 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 18:21:40.778888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.778877 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 18:21:40.779257 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.779246 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.779716 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.779693 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 18:21:40.779716 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.779714 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-62f7s\"" Jul 14 18:21:40.779888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.779747 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 18:21:40.779888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.779767 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 18:21:40.781446 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.781432 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kjlrj" Jul 14 18:21:40.781670 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.781653 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 18:21:40.782951 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.782936 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 18:21:40.783042 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.782961 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m22vx\"" Jul 14 18:21:40.783237 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.783223 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 18:21:40.783524 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.783505 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:21:40.783584 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.783534 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gmjgd\"" Jul 14 18:21:40.783584 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.783552 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 18:21:40.783679 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.783649 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h59zr" Jul 14 18:21:40.784422 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.784405 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 18:21:40.784505 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.784423 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-98sh2\"" Jul 14 18:21:40.784505 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.784408 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 18:21:40.784690 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.784676 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 18:21:40.785787 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.785773 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:40.785873 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:40.785838 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:21:40.786906 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.786889 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jtbl\" (UniqueName: \"kubernetes.io/projected/6c67705c-b9ce-4045-a31f-d58102ddc537-kube-api-access-2jtbl\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.786994 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.786918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-var-lib-openvswitch\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.786994 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.786939 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-run-openvswitch\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.786994 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.786961 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c42659ad-83fd-453f-afe1-39f0b9396257-ovnkube-config\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.786994 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.786985 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-sysctl-d\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.787130 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787019 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-sysctl-conf\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.787130 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787054 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-systemd\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.787130 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787081 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-slash\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.787130 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c42659ad-83fd-453f-afe1-39f0b9396257-ovn-node-metrics-cert\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.787250 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787144 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bdb299c-c772-4af9-8286-6fb4c2371885-tmp-dir\") pod \"node-resolver-tjpz8\" (UID: \"5bdb299c-c772-4af9-8286-6fb4c2371885\") " pod="openshift-dns/node-resolver-tjpz8" Jul 14 18:21:40.787250 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787162 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6261fa8-ee48-45f8-9576-3a75f2effa71-konnectivity-ca\") pod \"konnectivity-agent-66x25\" (UID: \"b6261fa8-ee48-45f8-9576-3a75f2effa71\") " pod="kube-system/konnectivity-agent-66x25" Jul 14 18:21:40.787250 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkjch\" (UniqueName: \"kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch\") pod \"network-check-target-rdlqb\" (UID: \"b5910206-776a-40cd-92ec-e5a154cc261a\") " pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:40.787250 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787198 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-modprobe-d\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.787250 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787227 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-sysconfig\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.787436 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787269 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-kubernetes\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.787436 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787307 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-host\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.787436 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787328 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c67705c-b9ce-4045-a31f-d58102ddc537-tmp\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.787436 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787343 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-var-lib-kubelet\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.787436 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-node-log\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.787436 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787373 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-log-socket\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.787436 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787393 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-cni-bin\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.787436 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787420 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-run\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.787436 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787436 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-etc-openvswitch\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.787843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787464 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c42659ad-83fd-453f-afe1-39f0b9396257-ovnkube-script-lib\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.787843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787505 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-kubelet\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.787843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787533 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-run-systemd\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.787843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787556 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c42659ad-83fd-453f-afe1-39f0b9396257-env-overrides\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.787843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787580 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5bdb299c-c772-4af9-8286-6fb4c2371885-hosts-file\") pod \"node-resolver-tjpz8\" (UID: \"5bdb299c-c772-4af9-8286-6fb4c2371885\") " pod="openshift-dns/node-resolver-tjpz8" Jul 14 18:21:40.787843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787611 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xljhk\" (UniqueName: \"kubernetes.io/projected/5bdb299c-c772-4af9-8286-6fb4c2371885-kube-api-access-xljhk\") pod \"node-resolver-tjpz8\" (UID: \"5bdb299c-c772-4af9-8286-6fb4c2371885\") " pod="openshift-dns/node-resolver-tjpz8" Jul 14 18:21:40.787843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787632 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-sys\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.787843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787656 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-lib-modules\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.787843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787679 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-tuned\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.787843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787708 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-cni-netd\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.787843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787732 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.787843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787756 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-systemd-units\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.787843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787795 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-run-netns\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.787843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787818 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-run-ovn\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.787843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787841 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-run-ovn-kubernetes\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.788409 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787863 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfnpb\" (UniqueName: \"kubernetes.io/projected/c42659ad-83fd-453f-afe1-39f0b9396257-kube-api-access-mfnpb\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.788409 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.787888 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6261fa8-ee48-45f8-9576-3a75f2effa71-agent-certs\") pod \"konnectivity-agent-66x25\" (UID: \"b6261fa8-ee48-45f8-9576-3a75f2effa71\") " pod="kube-system/konnectivity-agent-66x25" Jul 14 18:21:40.788409 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.788055 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.788931 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.788912 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 18:21:40.789008 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.788949 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6xhjz\"" Jul 14 18:21:40.789008 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.788980 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:21:40.790162 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.790149 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.791239 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.791222 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 18:21:40.791356 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.791263 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 18:21:40.791424 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.791369 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-66clj\"" Jul 14 18:21:40.791942 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.791928 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 18:21:40.792012 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.791954 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 18:21:40.792065 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.792025 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 18:21:40.792474 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.792459 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:40.792536 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:40.792517 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:21:40.792761 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.792747 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 18:21:40.793077 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.793063 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cr9kq\"" Jul 14 18:21:40.793244 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.793233 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 18:21:40.794735 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.794721 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.796353 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.796334 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 18:21:40.797621 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.797604 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 18:21:40.797745 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.797729 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pgp7r\"" Jul 14 18:21:40.797800 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.797784 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 18:21:40.798155 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.798140 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 18:21:40.823064 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.822571 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gnvkz" Jul 14 18:21:40.832774 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.832753 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gnvkz" Jul 14 18:21:40.878005 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.877987 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 18:21:40.888799 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.888780 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-cni-binary-copy\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.888877 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.888808 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c42659ad-83fd-453f-afe1-39f0b9396257-ovnkube-script-lib\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.888877 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.888825 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-device-dir\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.888877 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.888840 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ad7d617-55f3-43d9-8682-daf673a06f1c-system-cni-dir\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.888877 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.888859 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-dbus\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:40.889028 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.888882 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-run-systemd\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.889028 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.888902 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c42659ad-83fd-453f-afe1-39f0b9396257-env-overrides\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.889028 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.888918 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-socket-dir\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.889028 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.888933 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-registration-dir\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.889028 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.888949 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs\") pod \"network-metrics-daemon-29dsq\" (UID: \"2a487018-2ca6-4484-892b-8998a760a384\") " pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:40.889028 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.888955 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-run-systemd\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.889028 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.888971 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-multus-cni-dir\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.889028 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.888993 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-var-lib-kubelet\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889047 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5bdb299c-c772-4af9-8286-6fb4c2371885-hosts-file\") pod \"node-resolver-tjpz8\" (UID: \"5bdb299c-c772-4af9-8286-6fb4c2371885\") " pod="openshift-dns/node-resolver-tjpz8" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889111 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5bdb299c-c772-4af9-8286-6fb4c2371885-hosts-file\") pod \"node-resolver-tjpz8\" (UID: \"5bdb299c-c772-4af9-8286-6fb4c2371885\") " pod="openshift-dns/node-resolver-tjpz8" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889112 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfnpb\" (UniqueName: \"kubernetes.io/projected/c42659ad-83fd-453f-afe1-39f0b9396257-kube-api-access-mfnpb\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889141 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-sys-fs\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889155 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtcjs\" (UniqueName: \"kubernetes.io/projected/a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5-kube-api-access-mtcjs\") pod \"iptables-alerter-h59zr\" (UID: \"a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5\") " pod="openshift-network-operator/iptables-alerter-h59zr" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-systemd-units\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889201 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889209 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-systemd-units\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889234 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-run-ovn\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889269 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-run-ovn-kubernetes\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889274 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-run-ovn\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889315 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-system-cni-dir\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889323 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-run-ovn-kubernetes\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889341 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-run-k8s-cni-cncf-io\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.889360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889353 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c42659ad-83fd-453f-afe1-39f0b9396257-env-overrides\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6261fa8-ee48-45f8-9576-3a75f2effa71-agent-certs\") pod \"konnectivity-agent-66x25\" (UID: \"b6261fa8-ee48-45f8-9576-3a75f2effa71\") " pod="kube-system/konnectivity-agent-66x25" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889378 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c42659ad-83fd-453f-afe1-39f0b9396257-ovnkube-script-lib\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-run-openvswitch\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889423 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c42659ad-83fd-453f-afe1-39f0b9396257-ovnkube-config\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889450 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ad7d617-55f3-43d9-8682-daf673a06f1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889453 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-run-openvswitch\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-cnibin\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-sysctl-d\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889547 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-slash\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889572 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5-iptables-alerter-script\") pod \"iptables-alerter-h59zr\" (UID: \"a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5\") " pod="openshift-network-operator/iptables-alerter-h59zr" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889596 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bdb299c-c772-4af9-8286-6fb4c2371885-tmp-dir\") pod \"node-resolver-tjpz8\" (UID: \"5bdb299c-c772-4af9-8286-6fb4c2371885\") " pod="openshift-dns/node-resolver-tjpz8" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889620 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjch\" (UniqueName: \"kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch\") pod \"network-check-target-rdlqb\" (UID: \"b5910206-776a-40cd-92ec-e5a154cc261a\") " pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889642 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-modprobe-d\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889662 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-sysctl-d\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889667 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-kubernetes\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889619 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-slash\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889703 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c67705c-b9ce-4045-a31f-d58102ddc537-tmp\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.890031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889703 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889715 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-kubernetes\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889726 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-node-log\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889755 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-var-lib-cni-bin\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-kubelet-config\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889787 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-modprobe-d\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889805 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-var-lib-kubelet\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889834 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-log-socket\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889860 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-cni-bin\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889884 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-var-lib-kubelet\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889889 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b588j\" (UniqueName: \"kubernetes.io/projected/80ba9b7f-15c6-4fc0-82ab-8286c596aff1-kube-api-access-b588j\") pod \"node-ca-kjlrj\" (UID: \"80ba9b7f-15c6-4fc0-82ab-8286c596aff1\") " pod="openshift-image-registry/node-ca-kjlrj" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889887 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-node-log\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889891 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c42659ad-83fd-453f-afe1-39f0b9396257-ovnkube-config\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889915 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-hostroot\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889941 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-cni-bin\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889946 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-log-socket\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889925 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bdb299c-c772-4af9-8286-6fb4c2371885-tmp-dir\") pod \"node-resolver-tjpz8\" (UID: \"5bdb299c-c772-4af9-8286-6fb4c2371885\") " pod="openshift-dns/node-resolver-tjpz8" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889956 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5-host-slash\") pod \"iptables-alerter-h59zr\" (UID: \"a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5\") " pod="openshift-network-operator/iptables-alerter-h59zr" Jul 14 18:21:40.890725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.889997 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-run\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-etc-openvswitch\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890047 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-etc-selinux\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890061 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-etc-openvswitch\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890069 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ad7d617-55f3-43d9-8682-daf673a06f1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890088 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-multus-daemon-config\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890087 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-run\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-kubelet\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ad7d617-55f3-43d9-8682-daf673a06f1c-cnibin\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-multus-conf-dir\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890141 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-kubelet\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890154 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-etc-kubernetes\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890170 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwkk5\" (UniqueName: \"kubernetes.io/projected/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-kube-api-access-zwkk5\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xljhk\" (UniqueName: \"kubernetes.io/projected/5bdb299c-c772-4af9-8286-6fb4c2371885-kube-api-access-xljhk\") pod \"node-resolver-tjpz8\" (UID: \"5bdb299c-c772-4af9-8286-6fb4c2371885\") " pod="openshift-dns/node-resolver-tjpz8" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-sys\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-lib-modules\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.891392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890263 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-tuned\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890318 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-cni-netd\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890343 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80ba9b7f-15c6-4fc0-82ab-8286c596aff1-serviceca\") pod \"node-ca-kjlrj\" (UID: \"80ba9b7f-15c6-4fc0-82ab-8286c596aff1\") " pod="openshift-image-registry/node-ca-kjlrj" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890366 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ad7d617-55f3-43d9-8682-daf673a06f1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890390 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ad7d617-55f3-43d9-8682-daf673a06f1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890407 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-cni-netd\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890415 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-run-netns\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890416 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-lib-modules\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890451 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-run-netns\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890454 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-host-run-netns\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890488 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-var-lib-cni-multus\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890526 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890563 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-sys\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890581 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jtbl\" (UniqueName: \"kubernetes.io/projected/6c67705c-b9ce-4045-a31f-d58102ddc537-kube-api-access-2jtbl\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890604 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-var-lib-openvswitch\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890627 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkst7\" (UniqueName: \"kubernetes.io/projected/a8f80538-7175-4312-84ff-596e67768cab-kube-api-access-kkst7\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890643 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c42659ad-83fd-453f-afe1-39f0b9396257-var-lib-openvswitch\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.891888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890647 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-os-release\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-multus-socket-dir-parent\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-run-multus-certs\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890715 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-sysctl-conf\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890737 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-systemd\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c42659ad-83fd-453f-afe1-39f0b9396257-ovn-node-metrics-cert\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80ba9b7f-15c6-4fc0-82ab-8286c596aff1-host\") pod \"node-ca-kjlrj\" (UID: \"80ba9b7f-15c6-4fc0-82ab-8286c596aff1\") " pod="openshift-image-registry/node-ca-kjlrj" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890802 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-systemd\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890812 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br2lc\" (UniqueName: \"kubernetes.io/projected/2a487018-2ca6-4484-892b-8998a760a384-kube-api-access-br2lc\") pod \"network-metrics-daemon-29dsq\" (UID: \"2a487018-2ca6-4484-892b-8998a760a384\") " pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890841 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-sysctl-conf\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rdzs\" (UniqueName: \"kubernetes.io/projected/9ad7d617-55f3-43d9-8682-daf673a06f1c-kube-api-access-6rdzs\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890908 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6261fa8-ee48-45f8-9576-3a75f2effa71-konnectivity-ca\") pod \"konnectivity-agent-66x25\" (UID: \"b6261fa8-ee48-45f8-9576-3a75f2effa71\") " pod="kube-system/konnectivity-agent-66x25" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.890932 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-sysconfig\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.891031 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-host\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.891041 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-sysconfig\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.891055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9ad7d617-55f3-43d9-8682-daf673a06f1c-os-release\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.891085 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c67705c-b9ce-4045-a31f-d58102ddc537-host\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.892361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.891421 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6261fa8-ee48-45f8-9576-3a75f2effa71-konnectivity-ca\") pod \"konnectivity-agent-66x25\" (UID: \"b6261fa8-ee48-45f8-9576-3a75f2effa71\") " pod="kube-system/konnectivity-agent-66x25" Jul 14 18:21:40.892933 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.892915 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c42659ad-83fd-453f-afe1-39f0b9396257-ovn-node-metrics-cert\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.892988 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.892953 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c67705c-b9ce-4045-a31f-d58102ddc537-tmp\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.893043 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.893008 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6c67705c-b9ce-4045-a31f-d58102ddc537-etc-tuned\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.893043 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.893028 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6261fa8-ee48-45f8-9576-3a75f2effa71-agent-certs\") pod \"konnectivity-agent-66x25\" (UID: \"b6261fa8-ee48-45f8-9576-3a75f2effa71\") " pod="kube-system/konnectivity-agent-66x25" Jul 14 18:21:40.900391 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:40.900370 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:40.900391 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:40.900393 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:40.900536 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:40.900402 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fkjch for pod openshift-network-diagnostics/network-check-target-rdlqb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:40.900536 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:40.900460 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch podName:b5910206-776a-40cd-92ec-e5a154cc261a nodeName:}" failed. No retries permitted until 2026-07-14 18:21:41.400442809 +0000 UTC m=+2.937890594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fkjch" (UniqueName: "kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch") pod "network-check-target-rdlqb" (UID: "b5910206-776a-40cd-92ec-e5a154cc261a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:40.902938 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.902914 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jtbl\" (UniqueName: \"kubernetes.io/projected/6c67705c-b9ce-4045-a31f-d58102ddc537-kube-api-access-2jtbl\") pod \"tuned-7m6w6\" (UID: \"6c67705c-b9ce-4045-a31f-d58102ddc537\") " pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:40.903515 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.903492 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xljhk\" (UniqueName: \"kubernetes.io/projected/5bdb299c-c772-4af9-8286-6fb4c2371885-kube-api-access-xljhk\") pod \"node-resolver-tjpz8\" (UID: \"5bdb299c-c772-4af9-8286-6fb4c2371885\") " pod="openshift-dns/node-resolver-tjpz8" Jul 14 18:21:40.904029 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.904011 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfnpb\" (UniqueName: \"kubernetes.io/projected/c42659ad-83fd-453f-afe1-39f0b9396257-kube-api-access-mfnpb\") pod \"ovnkube-node-8pxlk\" (UID: \"c42659ad-83fd-453f-afe1-39f0b9396257\") " pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:40.991258 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991233 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80ba9b7f-15c6-4fc0-82ab-8286c596aff1-serviceca\") pod \"node-ca-kjlrj\" (UID: \"80ba9b7f-15c6-4fc0-82ab-8286c596aff1\") " pod="openshift-image-registry/node-ca-kjlrj" Jul 14 18:21:40.991258 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991261 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ad7d617-55f3-43d9-8682-daf673a06f1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.991451 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ad7d617-55f3-43d9-8682-daf673a06f1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.991451 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991316 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-run-netns\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.991451 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991339 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-var-lib-cni-multus\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.991451 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:40.991451 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkst7\" (UniqueName: \"kubernetes.io/projected/a8f80538-7175-4312-84ff-596e67768cab-kube-api-access-kkst7\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.991451 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:40.991450 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:40.991730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991457 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-var-lib-cni-multus\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.991730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991424 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-run-netns\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.991730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991461 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-os-release\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.991730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-multus-socket-dir-parent\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.991730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991516 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-os-release\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.991730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991533 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-run-multus-certs\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.991730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80ba9b7f-15c6-4fc0-82ab-8286c596aff1-host\") pod \"node-ca-kjlrj\" (UID: \"80ba9b7f-15c6-4fc0-82ab-8286c596aff1\") " pod="openshift-image-registry/node-ca-kjlrj" Jul 14 18:21:40.991730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991569 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-multus-socket-dir-parent\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.991730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991606 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80ba9b7f-15c6-4fc0-82ab-8286c596aff1-host\") pod \"node-ca-kjlrj\" (UID: \"80ba9b7f-15c6-4fc0-82ab-8286c596aff1\") " pod="openshift-image-registry/node-ca-kjlrj" Jul 14 18:21:40.991730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991612 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-run-multus-certs\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.991730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991646 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br2lc\" (UniqueName: \"kubernetes.io/projected/2a487018-2ca6-4484-892b-8998a760a384-kube-api-access-br2lc\") pod \"network-metrics-daemon-29dsq\" (UID: \"2a487018-2ca6-4484-892b-8998a760a384\") " pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:40.991730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991673 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rdzs\" (UniqueName: \"kubernetes.io/projected/9ad7d617-55f3-43d9-8682-daf673a06f1c-kube-api-access-6rdzs\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.991730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991702 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9ad7d617-55f3-43d9-8682-daf673a06f1c-os-release\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991753 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-cni-binary-copy\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-device-dir\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991828 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ad7d617-55f3-43d9-8682-daf673a06f1c-system-cni-dir\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991851 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-dbus\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991856 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9ad7d617-55f3-43d9-8682-daf673a06f1c-os-release\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-socket-dir\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991908 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-registration-dir\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991910 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ad7d617-55f3-43d9-8682-daf673a06f1c-cni-binary-copy\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991916 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ad7d617-55f3-43d9-8682-daf673a06f1c-system-cni-dir\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991926 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80ba9b7f-15c6-4fc0-82ab-8286c596aff1-serviceca\") pod \"node-ca-kjlrj\" (UID: \"80ba9b7f-15c6-4fc0-82ab-8286c596aff1\") " pod="openshift-image-registry/node-ca-kjlrj" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs\") pod \"network-metrics-daemon-29dsq\" (UID: \"2a487018-2ca6-4484-892b-8998a760a384\") " pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991967 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-multus-cni-dir\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.991973 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-device-dir\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992015 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-registration-dir\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992034 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-socket-dir\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992040 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-var-lib-kubelet\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992043 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-dbus\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:40.992336 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:40.992048 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992067 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-multus-cni-dir\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992073 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-var-lib-kubelet\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-sys-fs\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992109 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtcjs\" (UniqueName: \"kubernetes.io/projected/a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5-kube-api-access-mtcjs\") pod \"iptables-alerter-h59zr\" (UID: \"a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5\") " pod="openshift-network-operator/iptables-alerter-h59zr" Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992124 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-sys-fs\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:40.992129 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs podName:2a487018-2ca6-4484-892b-8998a760a384 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:41.492109348 +0000 UTC m=+3.029557128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs") pod "network-metrics-daemon-29dsq" (UID: "2a487018-2ca6-4484-892b-8998a760a384") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:40.992160 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret podName:0049cf44-b6fe-4c7c-8e53-adbd70b7eb46 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:41.492142816 +0000 UTC m=+3.029590783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret") pod "global-pull-secret-syncer-gm8z4" (UID: "0049cf44-b6fe-4c7c-8e53-adbd70b7eb46") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-system-cni-dir\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992218 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-run-k8s-cni-cncf-io\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992260 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-system-cni-dir\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992271 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ad7d617-55f3-43d9-8682-daf673a06f1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992312 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-cnibin\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992337 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-run-k8s-cni-cncf-io\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992339 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5-iptables-alerter-script\") pod \"iptables-alerter-h59zr\" (UID: \"a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5\") " pod="openshift-network-operator/iptables-alerter-h59zr" Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992348 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9ad7d617-55f3-43d9-8682-daf673a06f1c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.993031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992400 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-var-lib-cni-bin\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992404 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-cnibin\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-kubelet-config\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b588j\" (UniqueName: \"kubernetes.io/projected/80ba9b7f-15c6-4fc0-82ab-8286c596aff1-kube-api-access-b588j\") pod \"node-ca-kjlrj\" (UID: \"80ba9b7f-15c6-4fc0-82ab-8286c596aff1\") " pod="openshift-image-registry/node-ca-kjlrj" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992434 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-cni-binary-copy\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992446 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-host-var-lib-cni-bin\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992453 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-hostroot\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992446 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ad7d617-55f3-43d9-8682-daf673a06f1c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992483 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5-host-slash\") pod \"iptables-alerter-h59zr\" (UID: \"a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5\") " pod="openshift-network-operator/iptables-alerter-h59zr" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992495 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-kubelet-config\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992510 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992534 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-etc-selinux\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992551 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-hostroot\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992560 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ad7d617-55f3-43d9-8682-daf673a06f1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992576 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992586 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-multus-daemon-config\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992537 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5-host-slash\") pod \"iptables-alerter-h59zr\" (UID: \"a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5\") " pod="openshift-network-operator/iptables-alerter-h59zr" Jul 14 18:21:40.993511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992630 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8f80538-7175-4312-84ff-596e67768cab-etc-selinux\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:40.993959 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992684 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ad7d617-55f3-43d9-8682-daf673a06f1c-cnibin\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.993959 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992715 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-multus-conf-dir\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993959 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992739 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-etc-kubernetes\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993959 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992763 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwkk5\" (UniqueName: \"kubernetes.io/projected/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-kube-api-access-zwkk5\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993959 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992770 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ad7d617-55f3-43d9-8682-daf673a06f1c-cnibin\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:40.993959 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992797 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-multus-conf-dir\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993959 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992810 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-etc-kubernetes\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993959 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992837 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5-iptables-alerter-script\") pod \"iptables-alerter-h59zr\" (UID: \"a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5\") " pod="openshift-network-operator/iptables-alerter-h59zr" Jul 14 18:21:40.993959 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.992967 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-multus-daemon-config\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:40.993959 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:40.993029 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9ad7d617-55f3-43d9-8682-daf673a06f1c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:41.003390 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.003330 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b588j\" (UniqueName: \"kubernetes.io/projected/80ba9b7f-15c6-4fc0-82ab-8286c596aff1-kube-api-access-b588j\") pod \"node-ca-kjlrj\" (UID: \"80ba9b7f-15c6-4fc0-82ab-8286c596aff1\") " pod="openshift-image-registry/node-ca-kjlrj" Jul 14 18:21:41.005549 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.005525 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtcjs\" (UniqueName: \"kubernetes.io/projected/a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5-kube-api-access-mtcjs\") pod \"iptables-alerter-h59zr\" (UID: \"a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5\") " pod="openshift-network-operator/iptables-alerter-h59zr" Jul 14 18:21:41.005787 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.005763 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br2lc\" (UniqueName: \"kubernetes.io/projected/2a487018-2ca6-4484-892b-8998a760a384-kube-api-access-br2lc\") pod \"network-metrics-daemon-29dsq\" (UID: \"2a487018-2ca6-4484-892b-8998a760a384\") " pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:41.005861 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.005830 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkst7\" (UniqueName: \"kubernetes.io/projected/a8f80538-7175-4312-84ff-596e67768cab-kube-api-access-kkst7\") pod \"aws-ebs-csi-driver-node-wxwss\" (UID: \"a8f80538-7175-4312-84ff-596e67768cab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:41.006484 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.006462 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rdzs\" (UniqueName: \"kubernetes.io/projected/9ad7d617-55f3-43d9-8682-daf673a06f1c-kube-api-access-6rdzs\") pod \"multus-additional-cni-plugins-wpz5d\" (UID: \"9ad7d617-55f3-43d9-8682-daf673a06f1c\") " pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:41.006971 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.006951 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwkk5\" (UniqueName: \"kubernetes.io/projected/c9d9e2c3-b89f-41a4-932b-c3aad02404ee-kube-api-access-zwkk5\") pod \"multus-24f4c\" (UID: \"c9d9e2c3-b89f-41a4-932b-c3aad02404ee\") " pod="openshift-multus/multus-24f4c" Jul 14 18:21:41.118044 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.117954 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:21:41.133393 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.133369 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tjpz8" Jul 14 18:21:41.149055 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.149030 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-66x25" Jul 14 18:21:41.165690 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.165670 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" Jul 14 18:21:41.184262 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.184246 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kjlrj" Jul 14 18:21:41.187774 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.187757 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h59zr" Jul 14 18:21:41.225769 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.225751 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wpz5d" Jul 14 18:21:41.243339 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.243319 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-24f4c" Jul 14 18:21:41.247887 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.247871 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" Jul 14 18:21:41.298605 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.298586 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:41.496886 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.496819 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs\") pod \"network-metrics-daemon-29dsq\" (UID: \"2a487018-2ca6-4484-892b-8998a760a384\") " pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:41.496886 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.496858 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjch\" (UniqueName: \"kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch\") pod \"network-check-target-rdlqb\" (UID: \"b5910206-776a-40cd-92ec-e5a154cc261a\") " pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:41.497056 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:41.496963 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:41.497056 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:41.496975 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:41.497056 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:41.496991 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:41.497056 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:41.497004 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fkjch for pod openshift-network-diagnostics/network-check-target-rdlqb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:41.497056 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.497008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:41.497056 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:41.497032 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs podName:2a487018-2ca6-4484-892b-8998a760a384 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:42.497014099 +0000 UTC m=+4.034461877 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs") pod "network-metrics-daemon-29dsq" (UID: "2a487018-2ca6-4484-892b-8998a760a384") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:41.497313 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:41.497086 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch podName:b5910206-776a-40cd-92ec-e5a154cc261a nodeName:}" failed. No retries permitted until 2026-07-14 18:21:42.497065318 +0000 UTC m=+4.034513081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fkjch" (UniqueName: "kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch") pod "network-check-target-rdlqb" (UID: "b5910206-776a-40cd-92ec-e5a154cc261a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:41.497313 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:41.497102 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:41.497313 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:41.497158 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret podName:0049cf44-b6fe-4c7c-8e53-adbd70b7eb46 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:42.497143147 +0000 UTC m=+4.034590924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret") pod "global-pull-secret-syncer-gm8z4" (UID: "0049cf44-b6fe-4c7c-8e53-adbd70b7eb46") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:41.704463 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:41.704432 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42659ad_83fd_453f_afe1_39f0b9396257.slice/crio-0f55cb84782c7052805b0b4e25c3a88a47b5035fbd3faf07a035f0355fb6a304 WatchSource:0}: Error finding container 0f55cb84782c7052805b0b4e25c3a88a47b5035fbd3faf07a035f0355fb6a304: Status 404 returned error can't find the container with id 0f55cb84782c7052805b0b4e25c3a88a47b5035fbd3faf07a035f0355fb6a304 Jul 14 18:21:41.705641 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:41.705598 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bdb299c_c772_4af9_8286_6fb4c2371885.slice/crio-53abfed133b351fa74d0d6b8d8103be5421081bc896a4a2effe2931034ba2f5e WatchSource:0}: Error finding container 53abfed133b351fa74d0d6b8d8103be5421081bc896a4a2effe2931034ba2f5e: Status 404 returned error can't find the container with id 53abfed133b351fa74d0d6b8d8103be5421081bc896a4a2effe2931034ba2f5e Jul 14 18:21:41.713388 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.713360 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:21:41.724017 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:41.723995 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad7d617_55f3_43d9_8682_daf673a06f1c.slice/crio-c045666c0ffbb4a5263053d9b0282943f74ef351a9a326662d23ba2032667a4a WatchSource:0}: Error finding container c045666c0ffbb4a5263053d9b0282943f74ef351a9a326662d23ba2032667a4a: Status 404 returned error can't find the container with id c045666c0ffbb4a5263053d9b0282943f74ef351a9a326662d23ba2032667a4a Jul 14 18:21:41.734002 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:41.733982 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f80538_7175_4312_84ff_596e67768cab.slice/crio-4923ccd8a3ee8a7a3b188d40caf1152a0c08251cafa5585b528b2aea85200854 WatchSource:0}: Error finding container 4923ccd8a3ee8a7a3b188d40caf1152a0c08251cafa5585b528b2aea85200854: Status 404 returned error can't find the container with id 4923ccd8a3ee8a7a3b188d40caf1152a0c08251cafa5585b528b2aea85200854 Jul 14 18:21:41.754108 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:41.754083 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ba9b7f_15c6_4fc0_82ab_8286c596aff1.slice/crio-d34eabadbbf0a63e1c7b4b5920380f568adc88685be2dd72b43f0e6f49756796 WatchSource:0}: Error finding container d34eabadbbf0a63e1c7b4b5920380f568adc88685be2dd72b43f0e6f49756796: Status 404 returned error can't find the container with id d34eabadbbf0a63e1c7b4b5920380f568adc88685be2dd72b43f0e6f49756796 Jul 14 18:21:41.783501 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:41.783482 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6261fa8_ee48_45f8_9576_3a75f2effa71.slice/crio-63b52a7d43f882047c91a8156ae0b282e942e8b83caa6b8bd1fb1717ce6ba6f9 WatchSource:0}: Error finding container 63b52a7d43f882047c91a8156ae0b282e942e8b83caa6b8bd1fb1717ce6ba6f9: Status 404 returned error can't find the container with id 63b52a7d43f882047c91a8156ae0b282e942e8b83caa6b8bd1fb1717ce6ba6f9 Jul 14 18:21:41.835662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.835634 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 18:16:40 +0000 UTC" deadline="2028-05-01 02:21:57.517277874 +0000 UTC" Jul 14 18:21:41.835662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:41.835661 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15752h0m15.681619579s" Jul 14 18:21:41.914808 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:41.914787 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c67705c_b9ce_4045_a31f_d58102ddc537.slice/crio-8e8b93900b6775c241c5e4b3403c657caae25a2cf14a65a24dd095c12b018310 WatchSource:0}: Error finding container 8e8b93900b6775c241c5e4b3403c657caae25a2cf14a65a24dd095c12b018310: Status 404 returned error can't find the container with id 8e8b93900b6775c241c5e4b3403c657caae25a2cf14a65a24dd095c12b018310 Jul 14 18:21:41.952120 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:41.952093 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d9e2c3_b89f_41a4_932b_c3aad02404ee.slice/crio-6bba9636285f34e67d5a511c217b416aa920ac7ba672bb448b5e98200dd8e028 WatchSource:0}: Error finding container 6bba9636285f34e67d5a511c217b416aa920ac7ba672bb448b5e98200dd8e028: Status 404 returned error can't find the container with id 6bba9636285f34e67d5a511c217b416aa920ac7ba672bb448b5e98200dd8e028 Jul 14 18:21:42.215899 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.215812 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:42.216103 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:42.215962 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:21:42.216412 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.216389 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:42.216520 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:42.216495 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:21:42.216659 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.216560 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:42.216721 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:42.216661 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:21:42.221381 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.221332 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" event={"ID":"6c67705c-b9ce-4045-a31f-d58102ddc537","Type":"ContainerStarted","Data":"8e8b93900b6775c241c5e4b3403c657caae25a2cf14a65a24dd095c12b018310"} Jul 14 18:21:42.223189 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.223163 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-66x25" event={"ID":"b6261fa8-ee48-45f8-9576-3a75f2effa71","Type":"ContainerStarted","Data":"63b52a7d43f882047c91a8156ae0b282e942e8b83caa6b8bd1fb1717ce6ba6f9"} Jul 14 18:21:42.224372 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.224350 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kjlrj" event={"ID":"80ba9b7f-15c6-4fc0-82ab-8286c596aff1","Type":"ContainerStarted","Data":"d34eabadbbf0a63e1c7b4b5920380f568adc88685be2dd72b43f0e6f49756796"} Jul 14 18:21:42.225991 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.225964 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" event={"ID":"a8f80538-7175-4312-84ff-596e67768cab","Type":"ContainerStarted","Data":"4923ccd8a3ee8a7a3b188d40caf1152a0c08251cafa5585b528b2aea85200854"} Jul 14 18:21:42.227178 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.227154 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpz5d" event={"ID":"9ad7d617-55f3-43d9-8682-daf673a06f1c","Type":"ContainerStarted","Data":"c045666c0ffbb4a5263053d9b0282943f74ef351a9a326662d23ba2032667a4a"} Jul 14 18:21:42.228619 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.228596 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tjpz8" event={"ID":"5bdb299c-c772-4af9-8286-6fb4c2371885","Type":"ContainerStarted","Data":"53abfed133b351fa74d0d6b8d8103be5421081bc896a4a2effe2931034ba2f5e"} Jul 14 18:21:42.230972 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.230890 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" event={"ID":"c42659ad-83fd-453f-afe1-39f0b9396257","Type":"ContainerStarted","Data":"0f55cb84782c7052805b0b4e25c3a88a47b5035fbd3faf07a035f0355fb6a304"} Jul 14 18:21:42.232491 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.232445 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-24f4c" event={"ID":"c9d9e2c3-b89f-41a4-932b-c3aad02404ee","Type":"ContainerStarted","Data":"6bba9636285f34e67d5a511c217b416aa920ac7ba672bb448b5e98200dd8e028"} Jul 14 18:21:42.505675 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.505590 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs\") pod \"network-metrics-daemon-29dsq\" (UID: \"2a487018-2ca6-4484-892b-8998a760a384\") " pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:42.505675 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.505661 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjch\" (UniqueName: \"kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch\") pod \"network-check-target-rdlqb\" (UID: \"b5910206-776a-40cd-92ec-e5a154cc261a\") " pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:42.505894 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.505709 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:42.505894 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:42.505827 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:42.505894 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:42.505882 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret podName:0049cf44-b6fe-4c7c-8e53-adbd70b7eb46 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:44.505865563 +0000 UTC m=+6.043313329 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret") pod "global-pull-secret-syncer-gm8z4" (UID: "0049cf44-b6fe-4c7c-8e53-adbd70b7eb46") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:42.506279 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:42.506260 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:42.506375 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:42.506331 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs podName:2a487018-2ca6-4484-892b-8998a760a384 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:44.506313767 +0000 UTC m=+6.043761551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs") pod "network-metrics-daemon-29dsq" (UID: "2a487018-2ca6-4484-892b-8998a760a384") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:42.506443 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:42.506409 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:42.506443 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:42.506423 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:42.506443 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:42.506435 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fkjch for pod openshift-network-diagnostics/network-check-target-rdlqb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:42.506594 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:42.506476 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch podName:b5910206-776a-40cd-92ec-e5a154cc261a nodeName:}" failed. No retries permitted until 2026-07-14 18:21:44.506456743 +0000 UTC m=+6.043904509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fkjch" (UniqueName: "kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch") pod "network-check-target-rdlqb" (UID: "b5910206-776a-40cd-92ec-e5a154cc261a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:42.836454 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.836364 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 18:16:40 +0000 UTC" deadline="2028-02-08 11:07:27.449794415 +0000 UTC" Jul 14 18:21:42.836454 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:42.836406 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13768h45m44.613394813s" Jul 14 18:21:42.982149 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:21:42.982106 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b1f4d964eb8d8619f16afcc6da67af.slice/crio-df6bb318e1f5e2746b755ac989f6892c81c5e623bac9696fc884649d44b1b1fa WatchSource:0}: Error finding container df6bb318e1f5e2746b755ac989f6892c81c5e623bac9696fc884649d44b1b1fa: Status 404 returned error can't find the container with id df6bb318e1f5e2746b755ac989f6892c81c5e623bac9696fc884649d44b1b1fa Jul 14 18:21:43.237383 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:43.237343 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-224.ec2.internal" event={"ID":"13b1f4d964eb8d8619f16afcc6da67af","Type":"ContainerStarted","Data":"df6bb318e1f5e2746b755ac989f6892c81c5e623bac9696fc884649d44b1b1fa"} Jul 14 18:21:43.240561 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:43.240533 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal" event={"ID":"8f99cf49d21fb0224d8e1f094bc56ee0","Type":"ContainerStarted","Data":"3108de8150ec27818fc585ccadc721665fdc7d99f3077bbe3d83aa958465b993"} Jul 14 18:21:44.216338 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:44.216302 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:44.216811 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:44.216438 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:44.216811 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:44.216441 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:21:44.216811 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:44.216563 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:21:44.216811 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:44.216622 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:44.216811 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:44.216695 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:21:44.523949 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:44.523867 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:44.523949 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:44.523931 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs\") pod \"network-metrics-daemon-29dsq\" (UID: \"2a487018-2ca6-4484-892b-8998a760a384\") " pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:44.524170 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:44.523973 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjch\" (UniqueName: \"kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch\") pod \"network-check-target-rdlqb\" (UID: \"b5910206-776a-40cd-92ec-e5a154cc261a\") " pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:44.524170 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:44.524047 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:44.524170 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:44.524119 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret podName:0049cf44-b6fe-4c7c-8e53-adbd70b7eb46 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:48.524097421 +0000 UTC m=+10.061545198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret") pod "global-pull-secret-syncer-gm8z4" (UID: "0049cf44-b6fe-4c7c-8e53-adbd70b7eb46") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:44.524170 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:44.524122 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:44.524170 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:44.524139 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:44.524170 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:44.524151 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fkjch for pod openshift-network-diagnostics/network-check-target-rdlqb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:44.524499 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:44.524200 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch podName:b5910206-776a-40cd-92ec-e5a154cc261a nodeName:}" failed. No retries permitted until 2026-07-14 18:21:48.524185367 +0000 UTC m=+10.061633134 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fkjch" (UniqueName: "kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch") pod "network-check-target-rdlqb" (UID: "b5910206-776a-40cd-92ec-e5a154cc261a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:44.524499 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:44.524266 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:44.524499 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:44.524326 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs podName:2a487018-2ca6-4484-892b-8998a760a384 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:48.524312168 +0000 UTC m=+10.061759938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs") pod "network-metrics-daemon-29dsq" (UID: "2a487018-2ca6-4484-892b-8998a760a384") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:46.216203 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:46.216168 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:46.216666 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:46.216328 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:21:46.216894 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:46.216730 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:46.216894 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:46.216835 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:21:46.216894 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:46.216883 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:46.217111 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:46.216960 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:21:48.216239 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:48.216201 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:48.216239 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:48.216226 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:48.216775 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:48.216222 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:48.216775 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:48.216360 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:21:48.216775 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:48.216471 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:21:48.216775 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:48.216561 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:21:48.560081 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:48.560005 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs\") pod \"network-metrics-daemon-29dsq\" (UID: \"2a487018-2ca6-4484-892b-8998a760a384\") " pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:48.560081 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:48.560048 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjch\" (UniqueName: \"kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch\") pod \"network-check-target-rdlqb\" (UID: \"b5910206-776a-40cd-92ec-e5a154cc261a\") " pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:48.560081 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:48.560080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:48.560380 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:48.560169 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:48.560380 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:48.560194 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:48.560380 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:48.560214 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret podName:0049cf44-b6fe-4c7c-8e53-adbd70b7eb46 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:56.560201568 +0000 UTC m=+18.097649330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret") pod "global-pull-secret-syncer-gm8z4" (UID: "0049cf44-b6fe-4c7c-8e53-adbd70b7eb46") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:48.560380 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:48.560262 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs podName:2a487018-2ca6-4484-892b-8998a760a384 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:56.560243339 +0000 UTC m=+18.097691119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs") pod "network-metrics-daemon-29dsq" (UID: "2a487018-2ca6-4484-892b-8998a760a384") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:48.560380 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:48.560368 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:48.560664 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:48.560386 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:48.560664 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:48.560400 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fkjch for pod openshift-network-diagnostics/network-check-target-rdlqb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:48.560664 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:48.560445 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch podName:b5910206-776a-40cd-92ec-e5a154cc261a nodeName:}" failed. No retries permitted until 2026-07-14 18:21:56.560430329 +0000 UTC m=+18.097878094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fkjch" (UniqueName: "kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch") pod "network-check-target-rdlqb" (UID: "b5910206-776a-40cd-92ec-e5a154cc261a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:50.215966 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:50.215926 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:50.216458 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:50.215937 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:50.216458 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:50.216068 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:21:50.216458 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:50.216088 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:50.216458 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:50.216184 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:21:50.216458 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:50.216304 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:21:52.215491 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:52.215456 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:52.215491 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:52.215480 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:52.216013 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:52.215588 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:21:52.216013 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:52.215590 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:52.216013 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:52.215712 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:21:52.216013 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:52.215848 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:21:54.215535 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:54.215325 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:54.216020 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:54.215349 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:54.216020 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:54.215641 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:21:54.216020 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:54.215363 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:54.216020 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:54.215720 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:21:54.216020 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:54.215828 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:21:56.215995 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:56.215907 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:56.216431 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:56.215911 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:56.216431 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:56.216024 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:21:56.216431 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:56.215914 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:56.216431 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:56.216112 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:21:56.216431 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:56.216208 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:21:56.620389 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:56.620303 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjch\" (UniqueName: \"kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch\") pod \"network-check-target-rdlqb\" (UID: \"b5910206-776a-40cd-92ec-e5a154cc261a\") " pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:56.620389 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:56.620368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:56.620622 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:56.620404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs\") pod \"network-metrics-daemon-29dsq\" (UID: \"2a487018-2ca6-4484-892b-8998a760a384\") " pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:56.620622 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:56.620468 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:56.620622 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:56.620495 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:56.620622 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:56.620509 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fkjch for pod openshift-network-diagnostics/network-check-target-rdlqb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:56.620622 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:56.620520 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:56.620622 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:56.620530 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:56.620622 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:56.620570 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch podName:b5910206-776a-40cd-92ec-e5a154cc261a nodeName:}" failed. No retries permitted until 2026-07-14 18:22:12.620552626 +0000 UTC m=+34.158000405 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fkjch" (UniqueName: "kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch") pod "network-check-target-rdlqb" (UID: "b5910206-776a-40cd-92ec-e5a154cc261a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:56.620622 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:56.620591 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs podName:2a487018-2ca6-4484-892b-8998a760a384 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:12.620580685 +0000 UTC m=+34.158028458 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs") pod "network-metrics-daemon-29dsq" (UID: "2a487018-2ca6-4484-892b-8998a760a384") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:56.620622 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:56.620608 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret podName:0049cf44-b6fe-4c7c-8e53-adbd70b7eb46 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:12.620598618 +0000 UTC m=+34.158046384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret") pod "global-pull-secret-syncer-gm8z4" (UID: "0049cf44-b6fe-4c7c-8e53-adbd70b7eb46") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:58.216338 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:58.216302 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:21:58.216338 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:58.216332 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:21:58.216826 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:21:58.216332 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:21:58.216826 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:58.216441 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:21:58.216826 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:58.216517 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:21:58.216826 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:21:58.216587 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:22:00.215741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:00.215698 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:22:00.215741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:00.215741 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:22:00.216278 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:00.215749 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:22:00.216278 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:00.215842 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:22:00.216278 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:00.215963 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:22:00.216278 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:00.216030 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:22:01.499732 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:01.496761 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4182d42_0e6b_40dc_a7de_dcfe7dbd3bc5.slice/crio-d95e6712b4952f6aebe96182acbd7c656816804c70b95bb9e9db9ea833ac7570 WatchSource:0}: Error finding container d95e6712b4952f6aebe96182acbd7c656816804c70b95bb9e9db9ea833ac7570: Status 404 returned error can't find the container with id d95e6712b4952f6aebe96182acbd7c656816804c70b95bb9e9db9ea833ac7570 Jul 14 18:22:02.215686 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.215516 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:22:02.215776 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.215539 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:22:02.215846 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:02.215768 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:22:02.215901 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.215573 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:22:02.215901 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:02.215868 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:22:02.215968 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:02.215917 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:22:02.273902 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.273879 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-24f4c" event={"ID":"c9d9e2c3-b89f-41a4-932b-c3aad02404ee","Type":"ContainerStarted","Data":"72bdd8cc1e9484b491ee8ddbd0d5833a8c06d806fc3439fff3fec7383e8fc51f"} Jul 14 18:22:02.275270 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.275250 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" event={"ID":"6c67705c-b9ce-4045-a31f-d58102ddc537","Type":"ContainerStarted","Data":"940becb4ac187c19b1b2620750b461c9886a4dcaf27990dcdf0d67b9e7a2672f"} Jul 14 18:22:02.277491 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.276646 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-66x25" event={"ID":"b6261fa8-ee48-45f8-9576-3a75f2effa71","Type":"ContainerStarted","Data":"ed114d465e2910cb3c56ab26497d34b018a70171cd7a2cbe375178a0854df4e6"} Jul 14 18:22:02.280770 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.280739 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kjlrj" event={"ID":"80ba9b7f-15c6-4fc0-82ab-8286c596aff1","Type":"ContainerStarted","Data":"2f059eb85f5bcab8c84e2a73464b7ff409a9786a944b639bf2e81375f95c92eb"} Jul 14 18:22:02.283476 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.283452 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tjpz8" event={"ID":"5bdb299c-c772-4af9-8286-6fb4c2371885","Type":"ContainerStarted","Data":"f0eff56c5e8e52bd1d701f1789101d99852247872793c54fbeb3f5f7f4f1c4c8"} Jul 14 18:22:02.285568 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.285551 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:22:02.285891 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.285873 2573 generic.go:356] "Generic (PLEG): container finished" podID="c42659ad-83fd-453f-afe1-39f0b9396257" containerID="119897fd7050207981a28501f7604bc9598528358b2d76261c1dfb11c494c555" exitCode=1 Jul 14 18:22:02.285951 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.285931 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" event={"ID":"c42659ad-83fd-453f-afe1-39f0b9396257","Type":"ContainerStarted","Data":"81cc15aeb1f313d7be4219e48f361f39d89c1520365ab7af1d2a8c8502bfbe8d"} Jul 14 18:22:02.285992 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.285972 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" event={"ID":"c42659ad-83fd-453f-afe1-39f0b9396257","Type":"ContainerStarted","Data":"fd6f366ba2b3d2e50ec8fbdcdb27c0429fcb0f7658ee11c1f163069de55ab8c8"} Jul 14 18:22:02.286029 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.286000 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" event={"ID":"c42659ad-83fd-453f-afe1-39f0b9396257","Type":"ContainerDied","Data":"119897fd7050207981a28501f7604bc9598528358b2d76261c1dfb11c494c555"} Jul 14 18:22:02.286081 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.286029 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" event={"ID":"c42659ad-83fd-453f-afe1-39f0b9396257","Type":"ContainerStarted","Data":"00f49683fef4f300053eb5fdb025b620cd091f55326f69483b9925bb43911e76"} Jul 14 18:22:02.287340 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.287316 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-224.ec2.internal" event={"ID":"13b1f4d964eb8d8619f16afcc6da67af","Type":"ContainerStarted","Data":"b0ae8dbda399dad3f0d38eabeebd1049925f6cd1b6ac4499701156c3b4775f64"} Jul 14 18:22:02.288702 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.288683 2573 generic.go:356] "Generic (PLEG): container finished" podID="8f99cf49d21fb0224d8e1f094bc56ee0" containerID="b79cc4baf4724e8044a148b187da376a5be4babcb2179e3ec92ad6a9f74c865e" exitCode=0 Jul 14 18:22:02.288795 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.288749 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal" event={"ID":"8f99cf49d21fb0224d8e1f094bc56ee0","Type":"ContainerDied","Data":"b79cc4baf4724e8044a148b187da376a5be4babcb2179e3ec92ad6a9f74c865e"} Jul 14 18:22:02.290164 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.290141 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" event={"ID":"a8f80538-7175-4312-84ff-596e67768cab","Type":"ContainerStarted","Data":"2fd9f2f29e174038f1bc29e55eba5371d44b12362fafba2857d813178e1ea91e"} Jul 14 18:22:02.291528 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.291508 2573 generic.go:356] "Generic (PLEG): container finished" podID="9ad7d617-55f3-43d9-8682-daf673a06f1c" containerID="00573829694cb4674d998a68c732aee6d1ee97bb54b21ac6c213b6ec7a7edce1" exitCode=0 Jul 14 18:22:02.291605 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.291581 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpz5d" event={"ID":"9ad7d617-55f3-43d9-8682-daf673a06f1c","Type":"ContainerDied","Data":"00573829694cb4674d998a68c732aee6d1ee97bb54b21ac6c213b6ec7a7edce1"} Jul 14 18:22:02.292603 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.292579 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h59zr" event={"ID":"a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5","Type":"ContainerStarted","Data":"d95e6712b4952f6aebe96182acbd7c656816804c70b95bb9e9db9ea833ac7570"} Jul 14 18:22:02.299835 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.299799 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-24f4c" podStartSLOduration=3.7273467670000002 podStartE2EDuration="23.299787836s" totalImagesPullingTime="19.572441069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:02.299408833 +0000 UTC m=+23.836856617" watchObservedRunningTime="2026-07-14 18:22:02.299787836 +0000 UTC m=+23.837235620" Jul 14 18:22:02.329119 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.329083 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-224.ec2.internal" podStartSLOduration=22.329069827 podStartE2EDuration="22.329069827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:02.329012126 +0000 UTC m=+23.866459911" watchObservedRunningTime="2026-07-14 18:22:02.329069827 +0000 UTC m=+23.866517611" Jul 14 18:22:02.381938 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.381901 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tjpz8" podStartSLOduration=4.055778395 podStartE2EDuration="23.381886943s" totalImagesPullingTime="19.326108548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:02.347783941 +0000 UTC m=+23.885231725" watchObservedRunningTime="2026-07-14 18:22:02.381886943 +0000 UTC m=+23.919334732" Jul 14 18:22:02.382122 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.382104 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-66x25" podStartSLOduration=3.678061532 podStartE2EDuration="23.382097941s" totalImagesPullingTime="19.704036409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:02.381769906 +0000 UTC m=+23.919217692" watchObservedRunningTime="2026-07-14 18:22:02.382097941 +0000 UTC m=+23.919545726" Jul 14 18:22:02.458002 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:02.457954 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kjlrj" podStartSLOduration=3.721038779 podStartE2EDuration="23.457938373s" totalImagesPullingTime="19.736899594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:02.406221648 +0000 UTC m=+23.943669433" watchObservedRunningTime="2026-07-14 18:22:02.457938373 +0000 UTC m=+23.995386158" Jul 14 18:22:03.296827 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:03.296768 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:22:03.297537 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:03.297113 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" event={"ID":"c42659ad-83fd-453f-afe1-39f0b9396257","Type":"ContainerStarted","Data":"353cfa7fdb33e61b60ce011f4c59853aa2694cafe400d1be0cbc5d7dd581e8ad"} Jul 14 18:22:03.297537 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:03.297185 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" event={"ID":"c42659ad-83fd-453f-afe1-39f0b9396257","Type":"ContainerStarted","Data":"4aaf8374bf8d8ec1e6b2622a4cec9b212aa92547a58423e8cf3699b7f135a4d9"} Jul 14 18:22:03.298625 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:03.298593 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h59zr" event={"ID":"a4182d42-0e6b-40dc-a7de-dcfe7dbd3bc5","Type":"ContainerStarted","Data":"2716bfce37a86b81f76b23e1a9db15dccc3c47eca9ca9e6f60fb7512665ab055"} Jul 14 18:22:03.320267 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:03.320226 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7m6w6" podStartSLOduration=4.7765993479999995 podStartE2EDuration="24.320214466s" totalImagesPullingTime="19.543615118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:02.489213363 +0000 UTC m=+24.026661147" watchObservedRunningTime="2026-07-14 18:22:03.320214466 +0000 UTC m=+24.857662251" Jul 14 18:22:03.583860 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:03.583836 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 18:22:04.098646 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:04.098567 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T18:22:03.583855719Z","UUID":"251abc71-0ca9-49b7-bca6-ac5ab93337f2","Handler":null,"Name":"","Endpoint":""} Jul 14 18:22:04.101214 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:04.101179 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 18:22:04.101350 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:04.101223 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 18:22:04.215766 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:04.215736 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:22:04.215924 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:04.215841 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:22:04.215924 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:04.215844 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:22:04.215924 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:04.215864 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:22:04.216091 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:04.215946 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:22:04.216091 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:04.216044 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:22:04.302337 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:04.302300 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" event={"ID":"a8f80538-7175-4312-84ff-596e67768cab","Type":"ContainerStarted","Data":"063cf88fa0e8d16430715bc7e919818a7e4b45fda15f3e72c959c7527475c695"} Jul 14 18:22:04.304681 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:04.304655 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal" event={"ID":"8f99cf49d21fb0224d8e1f094bc56ee0","Type":"ContainerStarted","Data":"4ebdede002f30f836629bc5c3c1a0ecfa6e1750c66d6620ebfd3480cb97e67ad"} Jul 14 18:22:04.322181 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:04.322135 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-224.ec2.internal" podStartSLOduration=24.32211827 podStartE2EDuration="24.32211827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:04.322058181 +0000 UTC m=+25.859505966" watchObservedRunningTime="2026-07-14 18:22:04.32211827 +0000 UTC m=+25.859566057" Jul 14 18:22:04.322675 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:04.322647 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h59zr" podStartSLOduration=25.322637102 podStartE2EDuration="25.322637102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:03.339924708 +0000 UTC m=+24.877372492" watchObservedRunningTime="2026-07-14 18:22:04.322637102 +0000 UTC m=+25.860084933" Jul 14 18:22:05.309752 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:05.309603 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:22:05.310187 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:05.310129 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" event={"ID":"c42659ad-83fd-453f-afe1-39f0b9396257","Type":"ContainerStarted","Data":"4154abeee3440d66479453fb78a944f258502ccff7893cee4bbf140e6a474bbb"} Jul 14 18:22:05.312077 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:05.312053 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" event={"ID":"a8f80538-7175-4312-84ff-596e67768cab","Type":"ContainerStarted","Data":"b4f95d04ca3786ec97ff77e2a604206959ef573a466f11dcd84325194bbe0949"} Jul 14 18:22:05.330579 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:05.330542 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wxwss" podStartSLOduration=3.926468683 podStartE2EDuration="26.330531556s" totalImagesPullingTime="22.404062873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:05.330406928 +0000 UTC m=+26.867854712" watchObservedRunningTime="2026-07-14 18:22:05.330531556 +0000 UTC m=+26.867979341" Jul 14 18:22:06.215843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:06.215811 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:22:06.216060 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:06.215928 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:22:06.216060 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:06.215937 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:22:06.216060 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:06.215956 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:22:06.216060 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:06.216029 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:22:06.216304 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:06.216123 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:22:06.908069 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:06.907047 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-66x25" Jul 14 18:22:06.909672 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:06.908074 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-66x25" Jul 14 18:22:07.320791 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:07.320761 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:22:07.321224 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:07.321199 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" event={"ID":"c42659ad-83fd-453f-afe1-39f0b9396257","Type":"ContainerStarted","Data":"c8a379c289df1dab6d3b8182daa063d6c958d46394b2983ac9da20e792d3f969"} Jul 14 18:22:07.321551 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:07.321520 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-66x25" Jul 14 18:22:07.321834 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:07.321807 2573 scope.go:117] "RemoveContainer" containerID="119897fd7050207981a28501f7604bc9598528358b2d76261c1dfb11c494c555" Jul 14 18:22:07.322043 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:07.322023 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-66x25" Jul 14 18:22:08.215730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.215500 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:22:08.216275 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.215501 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:22:08.216275 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:08.215832 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:22:08.216275 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:08.215937 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:22:08.216275 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.215521 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:22:08.216275 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:08.216053 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:22:08.327589 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.327563 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:22:08.328088 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.328056 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" event={"ID":"c42659ad-83fd-453f-afe1-39f0b9396257","Type":"ContainerStarted","Data":"f1cb9dba4cc4e615bd6aaef68697d010430f8d1533d7429730936a14163d837d"} Jul 14 18:22:08.328222 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.328150 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:22:08.328278 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.328231 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:22:08.328443 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.328421 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:22:08.344937 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.344912 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:22:08.345045 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.345021 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:22:08.365006 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.364965 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" podStartSLOduration=9.347536255 podStartE2EDuration="29.36494681s" totalImagesPullingTime="20.017410555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:08.36347964 +0000 UTC m=+29.900927424" watchObservedRunningTime="2026-07-14 18:22:08.36494681 +0000 UTC m=+29.902394599" Jul 14 18:22:08.959579 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.959549 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29dsq"] Jul 14 18:22:08.959770 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.959688 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:22:08.959832 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:08.959806 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:22:08.961780 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.961752 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gm8z4"] Jul 14 18:22:08.961908 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.961841 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:22:08.961971 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:08.961931 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:22:08.968212 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.968188 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rdlqb"] Jul 14 18:22:08.968347 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:08.968277 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:22:08.968407 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:08.968381 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:22:10.215565 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:10.215529 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:22:10.215969 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:10.215628 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:22:10.215969 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:10.215537 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:22:10.215969 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:10.215654 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:22:10.215969 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:10.215735 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:22:10.215969 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:10.215802 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:22:10.333823 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:10.333789 2573 generic.go:356] "Generic (PLEG): container finished" podID="9ad7d617-55f3-43d9-8682-daf673a06f1c" containerID="45e4de44fd617a389638d2e1c8796512f0715b457510f72d3036b81993f65a31" exitCode=0 Jul 14 18:22:10.334000 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:10.333881 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpz5d" event={"ID":"9ad7d617-55f3-43d9-8682-daf673a06f1c","Type":"ContainerDied","Data":"45e4de44fd617a389638d2e1c8796512f0715b457510f72d3036b81993f65a31"} Jul 14 18:22:12.216224 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:12.216048 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:22:12.216627 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:12.216080 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:22:12.216627 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:12.216312 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:22:12.216627 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:12.216113 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:22:12.216627 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:12.216382 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:22:12.216627 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:12.216455 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:22:12.340347 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:12.340310 2573 generic.go:356] "Generic (PLEG): container finished" podID="9ad7d617-55f3-43d9-8682-daf673a06f1c" containerID="01cbe985da642375ffaccec3c2adc4d737d731d15a84cc9afbf7d664bb4857d8" exitCode=0 Jul 14 18:22:12.340498 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:12.340354 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpz5d" event={"ID":"9ad7d617-55f3-43d9-8682-daf673a06f1c","Type":"ContainerDied","Data":"01cbe985da642375ffaccec3c2adc4d737d731d15a84cc9afbf7d664bb4857d8"} Jul 14 18:22:12.644126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:12.644046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:22:12.644126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:12.644092 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs\") pod \"network-metrics-daemon-29dsq\" (UID: \"2a487018-2ca6-4484-892b-8998a760a384\") " pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:22:12.644126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:12.644117 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjch\" (UniqueName: \"kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch\") pod \"network-check-target-rdlqb\" (UID: \"b5910206-776a-40cd-92ec-e5a154cc261a\") " pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:22:12.644430 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:12.644200 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:22:12.644430 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:12.644273 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret podName:0049cf44-b6fe-4c7c-8e53-adbd70b7eb46 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:44.644254326 +0000 UTC m=+66.181702109 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret") pod "global-pull-secret-syncer-gm8z4" (UID: "0049cf44-b6fe-4c7c-8e53-adbd70b7eb46") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:22:12.644430 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:12.644199 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:22:12.644430 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:12.644344 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs podName:2a487018-2ca6-4484-892b-8998a760a384 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:44.64433252 +0000 UTC m=+66.181780283 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs") pod "network-metrics-daemon-29dsq" (UID: "2a487018-2ca6-4484-892b-8998a760a384") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:22:12.644430 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:12.644206 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:22:12.644430 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:12.644366 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:22:12.644430 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:12.644375 2573 projected.go:196] Error preparing data for projected volume kube-api-access-fkjch for pod openshift-network-diagnostics/network-check-target-rdlqb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:22:12.644430 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:12.644403 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch podName:b5910206-776a-40cd-92ec-e5a154cc261a nodeName:}" failed. No retries permitted until 2026-07-14 18:22:44.64439617 +0000 UTC m=+66.181843932 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fkjch" (UniqueName: "kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch") pod "network-check-target-rdlqb" (UID: "b5910206-776a-40cd-92ec-e5a154cc261a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:22:14.216079 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:14.216050 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:22:14.216518 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:14.216050 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:22:14.216518 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:14.216146 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gm8z4" podUID="0049cf44-b6fe-4c7c-8e53-adbd70b7eb46" Jul 14 18:22:14.216518 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:14.216059 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:22:14.216518 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:14.216227 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdlqb" podUID="b5910206-776a-40cd-92ec-e5a154cc261a" Jul 14 18:22:14.216518 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:14.216335 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29dsq" podUID="2a487018-2ca6-4484-892b-8998a760a384" Jul 14 18:22:14.346452 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:14.346425 2573 generic.go:356] "Generic (PLEG): container finished" podID="9ad7d617-55f3-43d9-8682-daf673a06f1c" containerID="284a392ede7b853a65499afe415ad9b9eeac97f698ac0d404bd67170c72969d8" exitCode=0 Jul 14 18:22:14.346592 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:14.346459 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpz5d" event={"ID":"9ad7d617-55f3-43d9-8682-daf673a06f1c","Type":"ContainerDied","Data":"284a392ede7b853a65499afe415ad9b9eeac97f698ac0d404bd67170c72969d8"} Jul 14 18:22:15.246786 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.246745 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-224.ec2.internal" event="NodeReady" Jul 14 18:22:15.247209 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.246912 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 18:22:15.292662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.292631 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-phr5m"] Jul 14 18:22:15.301182 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.301153 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68877f6bf8-lg2z8"] Jul 14 18:22:15.301335 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.301298 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-phr5m" Jul 14 18:22:15.308392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.308167 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 14 18:22:15.308512 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.308431 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:22:15.308512 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.308490 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-pfbtc\"" Jul 14 18:22:15.311061 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.311040 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-mwmlt"] Jul 14 18:22:15.311203 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.311172 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.318759 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.318741 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 18:22:15.319520 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.319501 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 18:22:15.319615 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.319505 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-psptv\"" Jul 14 18:22:15.328247 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.327729 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-59f484b746-8hs6t"] Jul 14 18:22:15.328247 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.327841 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:15.328808 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.328565 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 18:22:15.342240 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.342075 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 14 18:22:15.342583 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.342565 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lp766\"" Jul 14 18:22:15.342713 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.342623 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 14 18:22:15.342777 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.342723 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 18:22:15.342777 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.342729 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 14 18:22:15.342885 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.342676 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:22:15.346367 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.344324 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg"] Jul 14 18:22:15.347965 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.347939 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:15.355160 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.355136 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 14 18:22:15.355646 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.355627 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 18:22:15.356003 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.355796 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 14 18:22:15.356333 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.356312 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 14 18:22:15.356548 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.356530 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 14 18:22:15.356662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.356645 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 18:22:15.356732 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.356601 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xxw7v\"" Jul 14 18:22:15.356791 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.356552 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 14 18:22:15.364423 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.364399 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-tkrhd"] Jul 14 18:22:15.364535 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.364519 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" Jul 14 18:22:15.368358 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.368216 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 14 18:22:15.368455 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.368361 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:22:15.368723 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.368698 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 14 18:22:15.368862 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.368836 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kqnxg\"" Jul 14 18:22:15.369260 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.369088 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 14 18:22:15.379957 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.379938 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-mwmlt"] Jul 14 18:22:15.380060 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.379972 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-phr5m"] Jul 14 18:22:15.380060 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.379994 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg"] Jul 14 18:22:15.380060 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.380017 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb"] Jul 14 18:22:15.380217 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.380062 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.384472 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.384454 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 14 18:22:15.384568 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.384484 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 18:22:15.385303 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.385270 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 14 18:22:15.385386 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.385374 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-b6w7n\"" Jul 14 18:22:15.385623 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.385588 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 18:22:15.390380 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.390360 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 14 18:22:15.396239 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.396204 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8"] Jul 14 18:22:15.396381 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.396364 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:15.399148 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.399128 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 14 18:22:15.399477 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.399459 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dz5cv\"" Jul 14 18:22:15.399579 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.399558 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 18:22:15.401405 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.401384 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 14 18:22:15.401686 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.401670 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 18:22:15.410217 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.410197 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-jcfxl"] Jul 14 18:22:15.410375 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.410355 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" Jul 14 18:22:15.413337 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.413315 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-lszrk\"" Jul 14 18:22:15.413337 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.413329 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 14 18:22:15.413813 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.413798 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 14 18:22:15.414180 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.414160 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 14 18:22:15.414315 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.414186 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:22:15.431518 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.431494 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr"] Jul 14 18:22:15.431645 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.431579 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jcfxl" Jul 14 18:22:15.437082 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.437057 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 18:22:15.437362 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.437339 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rglvm\"" Jul 14 18:22:15.438133 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.438114 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 18:22:15.450222 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.450200 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-tkrhd"] Jul 14 18:22:15.450354 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.450229 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68877f6bf8-lg2z8"] Jul 14 18:22:15.450354 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.450250 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59f484b746-8hs6t"] Jul 14 18:22:15.450354 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.450271 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb"] Jul 14 18:22:15.450354 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.450310 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-jcfxl"] Jul 14 18:22:15.450354 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.450329 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr"] Jul 14 18:22:15.450354 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.450340 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" Jul 14 18:22:15.450354 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.450352 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8"] Jul 14 18:22:15.450677 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.450380 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hjjpz"] Jul 14 18:22:15.453723 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.453707 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 14 18:22:15.455240 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.455220 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 14 18:22:15.455342 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.455222 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:22:15.455410 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.455352 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-sd5kk\"" Jul 14 18:22:15.463848 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.463827 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.463959 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.463879 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/159e6d97-7ed3-42d8-b502-9add9b40eccb-config\") pod \"console-operator-7866497494-mwmlt\" (UID: \"159e6d97-7ed3-42d8-b502-9add9b40eccb\") " pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:15.463959 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.463909 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4lq7\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-kube-api-access-r4lq7\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.464049 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.463960 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bsj2\" (UniqueName: \"kubernetes.io/projected/159e6d97-7ed3-42d8-b502-9add9b40eccb-kube-api-access-5bsj2\") pod \"console-operator-7866497494-mwmlt\" (UID: \"159e6d97-7ed3-42d8-b502-9add9b40eccb\") " pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:15.464049 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.463992 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:15.464049 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464011 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:15.464222 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464070 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-default-certificate\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:15.464222 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464091 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6392abcd-2ccd-410b-8c10-08a58197237e-tmp\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.464222 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-bound-sa-token\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.464222 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464139 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e730311-d011-495f-a4c7-a0097413011c-ca-trust-extracted\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.464222 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464155 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e730311-d011-495f-a4c7-a0097413011c-registry-certificates\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.464222 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6392abcd-2ccd-410b-8c10-08a58197237e-snapshots\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.464470 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464239 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e730311-d011-495f-a4c7-a0097413011c-image-registry-private-configuration\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.464470 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464336 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6392abcd-2ccd-410b-8c10-08a58197237e-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.464470 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464357 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkdrz\" (UniqueName: \"kubernetes.io/projected/d1f7e1f6-6948-4d4d-be15-eca40f74051f-kube-api-access-xkdrz\") pod \"volume-data-source-validator-78495597f6-phr5m\" (UID: \"d1f7e1f6-6948-4d4d-be15-eca40f74051f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-phr5m" Jul 14 18:22:15.464470 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464377 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/113b1fbe-22b0-4d6f-8f4f-ede50c3b9242-config\") pod \"service-ca-operator-7695fc6c8c-m94bg\" (UID: \"113b1fbe-22b0-4d6f-8f4f-ede50c3b9242\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" Jul 14 18:22:15.464470 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e730311-d011-495f-a4c7-a0097413011c-installation-pull-secrets\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.464470 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464437 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6392abcd-2ccd-410b-8c10-08a58197237e-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.464470 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464459 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpjqk\" (UniqueName: \"kubernetes.io/projected/113b1fbe-22b0-4d6f-8f4f-ede50c3b9242-kube-api-access-zpjqk\") pod \"service-ca-operator-7695fc6c8c-m94bg\" (UID: \"113b1fbe-22b0-4d6f-8f4f-ede50c3b9242\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" Jul 14 18:22:15.464743 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/159e6d97-7ed3-42d8-b502-9add9b40eccb-serving-cert\") pod \"console-operator-7866497494-mwmlt\" (UID: \"159e6d97-7ed3-42d8-b502-9add9b40eccb\") " pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:15.464743 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464522 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e730311-d011-495f-a4c7-a0097413011c-trusted-ca\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.464743 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464554 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h9ht\" (UniqueName: \"kubernetes.io/projected/6392abcd-2ccd-410b-8c10-08a58197237e-kube-api-access-4h9ht\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.464743 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464575 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6392abcd-2ccd-410b-8c10-08a58197237e-serving-cert\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.464743 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464599 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/113b1fbe-22b0-4d6f-8f4f-ede50c3b9242-serving-cert\") pod \"service-ca-operator-7695fc6c8c-m94bg\" (UID: \"113b1fbe-22b0-4d6f-8f4f-ede50c3b9242\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" Jul 14 18:22:15.464743 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464614 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-stats-auth\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:15.464743 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464639 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hgjp\" (UniqueName: \"kubernetes.io/projected/c7cf164f-807a-4191-acdd-e7d5f3c6de75-kube-api-access-9hgjp\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:15.464743 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.464674 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/159e6d97-7ed3-42d8-b502-9add9b40eccb-trusted-ca\") pod \"console-operator-7866497494-mwmlt\" (UID: \"159e6d97-7ed3-42d8-b502-9add9b40eccb\") " pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:15.476321 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.476280 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjjpz"] Jul 14 18:22:15.476409 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.476342 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7v79r"] Jul 14 18:22:15.476409 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.476407 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjjpz" Jul 14 18:22:15.480017 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.479973 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-58lnk\"" Jul 14 18:22:15.480233 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.480215 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 18:22:15.480332 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.480298 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 18:22:15.480494 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.480468 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 18:22:15.493000 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.492981 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7v79r"] Jul 14 18:22:15.493094 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.493075 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:15.497261 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.497115 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 18:22:15.497261 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.497174 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wk6vb\"" Jul 14 18:22:15.497452 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.497423 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 18:22:15.565705 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.565678 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krr46\" (UniqueName: \"kubernetes.io/projected/42555e5b-8ae2-4677-a74a-bbacb3393030-kube-api-access-krr46\") pod \"kube-storage-version-migrator-operator-799cb7f958-n22g8\" (UID: \"42555e5b-8ae2-4677-a74a-bbacb3393030\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" Jul 14 18:22:15.565887 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.565712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ppdj\" (UniqueName: \"kubernetes.io/projected/af335ced-5db5-4190-832e-8ad80e24915b-kube-api-access-6ppdj\") pod \"cluster-samples-operator-65db858894-bhjwr\" (UID: \"af335ced-5db5-4190-832e-8ad80e24915b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" Jul 14 18:22:15.565887 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.565756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-default-certificate\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:15.565887 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.565802 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6392abcd-2ccd-410b-8c10-08a58197237e-tmp\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.566051 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.565983 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-bound-sa-token\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.566051 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566021 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e730311-d011-495f-a4c7-a0097413011c-ca-trust-extracted\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.566133 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566050 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e730311-d011-495f-a4c7-a0097413011c-registry-certificates\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.566133 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566080 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/24acf903-caba-43ed-bd46-cf61a79524c9-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-l4fpb\" (UID: \"24acf903-caba-43ed-bd46-cf61a79524c9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:15.566133 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566111 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6392abcd-2ccd-410b-8c10-08a58197237e-snapshots\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.566253 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e730311-d011-495f-a4c7-a0097413011c-image-registry-private-configuration\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.566253 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6392abcd-2ccd-410b-8c10-08a58197237e-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.566253 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566206 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkdrz\" (UniqueName: \"kubernetes.io/projected/d1f7e1f6-6948-4d4d-be15-eca40f74051f-kube-api-access-xkdrz\") pod \"volume-data-source-validator-78495597f6-phr5m\" (UID: \"d1f7e1f6-6948-4d4d-be15-eca40f74051f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-phr5m" Jul 14 18:22:15.566253 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566231 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67d27\" (UniqueName: \"kubernetes.io/projected/24acf903-caba-43ed-bd46-cf61a79524c9-kube-api-access-67d27\") pod \"cluster-monitoring-operator-5f7f494dd4-l4fpb\" (UID: \"24acf903-caba-43ed-bd46-cf61a79524c9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:15.566455 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/113b1fbe-22b0-4d6f-8f4f-ede50c3b9242-config\") pod \"service-ca-operator-7695fc6c8c-m94bg\" (UID: \"113b1fbe-22b0-4d6f-8f4f-ede50c3b9242\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" Jul 14 18:22:15.566455 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566334 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e730311-d011-495f-a4c7-a0097413011c-installation-pull-secrets\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.566455 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566344 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6392abcd-2ccd-410b-8c10-08a58197237e-tmp\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.566561 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566519 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e730311-d011-495f-a4c7-a0097413011c-ca-trust-extracted\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.567741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566352 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6392abcd-2ccd-410b-8c10-08a58197237e-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.567741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566666 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42555e5b-8ae2-4677-a74a-bbacb3393030-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-n22g8\" (UID: \"42555e5b-8ae2-4677-a74a-bbacb3393030\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" Jul 14 18:22:15.567741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42555e5b-8ae2-4677-a74a-bbacb3393030-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-n22g8\" (UID: \"42555e5b-8ae2-4677-a74a-bbacb3393030\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" Jul 14 18:22:15.567741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566757 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6392abcd-2ccd-410b-8c10-08a58197237e-snapshots\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.567741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566758 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpjqk\" (UniqueName: \"kubernetes.io/projected/113b1fbe-22b0-4d6f-8f4f-ede50c3b9242-kube-api-access-zpjqk\") pod \"service-ca-operator-7695fc6c8c-m94bg\" (UID: \"113b1fbe-22b0-4d6f-8f4f-ede50c3b9242\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" Jul 14 18:22:15.567741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566799 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/159e6d97-7ed3-42d8-b502-9add9b40eccb-serving-cert\") pod \"console-operator-7866497494-mwmlt\" (UID: \"159e6d97-7ed3-42d8-b502-9add9b40eccb\") " pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:15.567741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566834 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e730311-d011-495f-a4c7-a0097413011c-trusted-ca\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.567741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566863 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h9ht\" (UniqueName: \"kubernetes.io/projected/6392abcd-2ccd-410b-8c10-08a58197237e-kube-api-access-4h9ht\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.567741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566897 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6392abcd-2ccd-410b-8c10-08a58197237e-serving-cert\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.567741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/113b1fbe-22b0-4d6f-8f4f-ede50c3b9242-serving-cert\") pod \"service-ca-operator-7695fc6c8c-m94bg\" (UID: \"113b1fbe-22b0-4d6f-8f4f-ede50c3b9242\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" Jul 14 18:22:15.567741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566954 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-stats-auth\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:15.567741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.566981 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e2729e0-7391-4488-8c08-285392a648be-cert\") pod \"ingress-canary-hjjpz\" (UID: \"8e2729e0-7391-4488-8c08-285392a648be\") " pod="openshift-ingress-canary/ingress-canary-hjjpz" Jul 14 18:22:15.567741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.567016 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-l4fpb\" (UID: \"24acf903-caba-43ed-bd46-cf61a79524c9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:15.567741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.567049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hgjp\" (UniqueName: \"kubernetes.io/projected/c7cf164f-807a-4191-acdd-e7d5f3c6de75-kube-api-access-9hgjp\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:15.567741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.567071 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e730311-d011-495f-a4c7-a0097413011c-registry-certificates\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.567079 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/159e6d97-7ed3-42d8-b502-9add9b40eccb-trusted-ca\") pod \"console-operator-7866497494-mwmlt\" (UID: \"159e6d97-7ed3-42d8-b502-9add9b40eccb\") " pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.567123 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.567233 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6392abcd-2ccd-410b-8c10-08a58197237e-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.567315 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/113b1fbe-22b0-4d6f-8f4f-ede50c3b9242-config\") pod \"service-ca-operator-7695fc6c8c-m94bg\" (UID: \"113b1fbe-22b0-4d6f-8f4f-ede50c3b9242\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:15.567338 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:15.567353 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68877f6bf8-lg2z8: secret "image-registry-tls" not found Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.567383 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/159e6d97-7ed3-42d8-b502-9add9b40eccb-config\") pod \"console-operator-7866497494-mwmlt\" (UID: \"159e6d97-7ed3-42d8-b502-9add9b40eccb\") " pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:15.567416 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls podName:4e730311-d011-495f-a4c7-a0097413011c nodeName:}" failed. No retries permitted until 2026-07-14 18:22:16.067397014 +0000 UTC m=+37.604844780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls") pod "image-registry-68877f6bf8-lg2z8" (UID: "4e730311-d011-495f-a4c7-a0097413011c") : secret "image-registry-tls" not found Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.567444 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4lq7\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-kube-api-access-r4lq7\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.567467 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6392abcd-2ccd-410b-8c10-08a58197237e-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.567475 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9phc\" (UniqueName: \"kubernetes.io/projected/3c17b6fe-41a0-4b30-8dbc-8a9bcf472966-kube-api-access-q9phc\") pod \"network-check-source-869dcfcdcd-jcfxl\" (UID: \"3c17b6fe-41a0-4b30-8dbc-8a9bcf472966\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jcfxl" Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.567543 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bsj2\" (UniqueName: \"kubernetes.io/projected/159e6d97-7ed3-42d8-b502-9add9b40eccb-kube-api-access-5bsj2\") pod \"console-operator-7866497494-mwmlt\" (UID: \"159e6d97-7ed3-42d8-b502-9add9b40eccb\") " pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.567569 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.567596 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:15.567665 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 18:22:15.568574 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:15.567717 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs podName:c7cf164f-807a-4191-acdd-e7d5f3c6de75 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:16.067703918 +0000 UTC m=+37.605151684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs") pod "router-default-59f484b746-8hs6t" (UID: "c7cf164f-807a-4191-acdd-e7d5f3c6de75") : secret "router-metrics-certs-default" not found Jul 14 18:22:15.569392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.568560 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/159e6d97-7ed3-42d8-b502-9add9b40eccb-trusted-ca\") pod \"console-operator-7866497494-mwmlt\" (UID: \"159e6d97-7ed3-42d8-b502-9add9b40eccb\") " pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:15.569392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.569109 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/159e6d97-7ed3-42d8-b502-9add9b40eccb-config\") pod \"console-operator-7866497494-mwmlt\" (UID: \"159e6d97-7ed3-42d8-b502-9add9b40eccb\") " pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:15.569392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.569194 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e730311-d011-495f-a4c7-a0097413011c-trusted-ca\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.569392 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:15.569228 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle podName:c7cf164f-807a-4191-acdd-e7d5f3c6de75 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:16.069212016 +0000 UTC m=+37.606659804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle") pod "router-default-59f484b746-8hs6t" (UID: "c7cf164f-807a-4191-acdd-e7d5f3c6de75") : configmap references non-existent config key: service-ca.crt Jul 14 18:22:15.569392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.569356 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bqj\" (UniqueName: \"kubernetes.io/projected/8e2729e0-7391-4488-8c08-285392a648be-kube-api-access-d8bqj\") pod \"ingress-canary-hjjpz\" (UID: \"8e2729e0-7391-4488-8c08-285392a648be\") " pod="openshift-ingress-canary/ingress-canary-hjjpz" Jul 14 18:22:15.569644 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.569515 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bhjwr\" (UID: \"af335ced-5db5-4190-832e-8ad80e24915b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" Jul 14 18:22:15.571620 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.571590 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6392abcd-2ccd-410b-8c10-08a58197237e-serving-cert\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.571727 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.571625 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e730311-d011-495f-a4c7-a0097413011c-installation-pull-secrets\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.571727 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.571673 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-default-certificate\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:15.571854 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.571817 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/113b1fbe-22b0-4d6f-8f4f-ede50c3b9242-serving-cert\") pod \"service-ca-operator-7695fc6c8c-m94bg\" (UID: \"113b1fbe-22b0-4d6f-8f4f-ede50c3b9242\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" Jul 14 18:22:15.576167 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.576142 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e730311-d011-495f-a4c7-a0097413011c-image-registry-private-configuration\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.576312 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.576184 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-stats-auth\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:15.576929 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.576908 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/159e6d97-7ed3-42d8-b502-9add9b40eccb-serving-cert\") pod \"console-operator-7866497494-mwmlt\" (UID: \"159e6d97-7ed3-42d8-b502-9add9b40eccb\") " pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:15.578730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.578711 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-bound-sa-token\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.582586 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.582565 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpjqk\" (UniqueName: \"kubernetes.io/projected/113b1fbe-22b0-4d6f-8f4f-ede50c3b9242-kube-api-access-zpjqk\") pod \"service-ca-operator-7695fc6c8c-m94bg\" (UID: \"113b1fbe-22b0-4d6f-8f4f-ede50c3b9242\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" Jul 14 18:22:15.583252 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.583229 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h9ht\" (UniqueName: \"kubernetes.io/projected/6392abcd-2ccd-410b-8c10-08a58197237e-kube-api-access-4h9ht\") pod \"insights-operator-68d9dd94cf-tkrhd\" (UID: \"6392abcd-2ccd-410b-8c10-08a58197237e\") " pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.583654 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.583612 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkdrz\" (UniqueName: \"kubernetes.io/projected/d1f7e1f6-6948-4d4d-be15-eca40f74051f-kube-api-access-xkdrz\") pod \"volume-data-source-validator-78495597f6-phr5m\" (UID: \"d1f7e1f6-6948-4d4d-be15-eca40f74051f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-phr5m" Jul 14 18:22:15.585165 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.585127 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bsj2\" (UniqueName: \"kubernetes.io/projected/159e6d97-7ed3-42d8-b502-9add9b40eccb-kube-api-access-5bsj2\") pod \"console-operator-7866497494-mwmlt\" (UID: \"159e6d97-7ed3-42d8-b502-9add9b40eccb\") " pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:15.585806 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.585783 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hgjp\" (UniqueName: \"kubernetes.io/projected/c7cf164f-807a-4191-acdd-e7d5f3c6de75-kube-api-access-9hgjp\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:15.586160 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.586138 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4lq7\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-kube-api-access-r4lq7\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:15.611318 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.611276 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-phr5m" Jul 14 18:22:15.639315 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.639277 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:15.670908 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.670877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:15.671042 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.670951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67d27\" (UniqueName: \"kubernetes.io/projected/24acf903-caba-43ed-bd46-cf61a79524c9-kube-api-access-67d27\") pod \"cluster-monitoring-operator-5f7f494dd4-l4fpb\" (UID: \"24acf903-caba-43ed-bd46-cf61a79524c9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:15.671042 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.671026 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42555e5b-8ae2-4677-a74a-bbacb3393030-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-n22g8\" (UID: \"42555e5b-8ae2-4677-a74a-bbacb3393030\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" Jul 14 18:22:15.671157 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.671053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42555e5b-8ae2-4677-a74a-bbacb3393030-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-n22g8\" (UID: \"42555e5b-8ae2-4677-a74a-bbacb3393030\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" Jul 14 18:22:15.671157 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.671124 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e2729e0-7391-4488-8c08-285392a648be-cert\") pod \"ingress-canary-hjjpz\" (UID: \"8e2729e0-7391-4488-8c08-285392a648be\") " pod="openshift-ingress-canary/ingress-canary-hjjpz" Jul 14 18:22:15.671157 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.671152 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-l4fpb\" (UID: \"24acf903-caba-43ed-bd46-cf61a79524c9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:15.671391 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.671208 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2tsq\" (UniqueName: \"kubernetes.io/projected/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-kube-api-access-r2tsq\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:15.671391 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.671268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9phc\" (UniqueName: \"kubernetes.io/projected/3c17b6fe-41a0-4b30-8dbc-8a9bcf472966-kube-api-access-q9phc\") pod \"network-check-source-869dcfcdcd-jcfxl\" (UID: \"3c17b6fe-41a0-4b30-8dbc-8a9bcf472966\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jcfxl" Jul 14 18:22:15.671391 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.671348 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-tmp-dir\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:15.671575 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.671404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bqj\" (UniqueName: \"kubernetes.io/projected/8e2729e0-7391-4488-8c08-285392a648be-kube-api-access-d8bqj\") pod \"ingress-canary-hjjpz\" (UID: \"8e2729e0-7391-4488-8c08-285392a648be\") " pod="openshift-ingress-canary/ingress-canary-hjjpz" Jul 14 18:22:15.671575 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.671438 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bhjwr\" (UID: \"af335ced-5db5-4190-832e-8ad80e24915b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" Jul 14 18:22:15.671575 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.671467 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krr46\" (UniqueName: \"kubernetes.io/projected/42555e5b-8ae2-4677-a74a-bbacb3393030-kube-api-access-krr46\") pod \"kube-storage-version-migrator-operator-799cb7f958-n22g8\" (UID: \"42555e5b-8ae2-4677-a74a-bbacb3393030\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" Jul 14 18:22:15.671575 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.671504 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ppdj\" (UniqueName: \"kubernetes.io/projected/af335ced-5db5-4190-832e-8ad80e24915b-kube-api-access-6ppdj\") pod \"cluster-samples-operator-65db858894-bhjwr\" (UID: \"af335ced-5db5-4190-832e-8ad80e24915b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" Jul 14 18:22:15.671575 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.671573 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-config-volume\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:15.671840 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.671616 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/24acf903-caba-43ed-bd46-cf61a79524c9-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-l4fpb\" (UID: \"24acf903-caba-43ed-bd46-cf61a79524c9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:15.672458 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.672427 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/24acf903-caba-43ed-bd46-cf61a79524c9-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-l4fpb\" (UID: \"24acf903-caba-43ed-bd46-cf61a79524c9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:15.672570 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:15.672536 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 18:22:15.672626 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:15.672612 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls podName:24acf903-caba-43ed-bd46-cf61a79524c9 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:16.172595247 +0000 UTC m=+37.710043015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-l4fpb" (UID: "24acf903-caba-43ed-bd46-cf61a79524c9") : secret "cluster-monitoring-operator-tls" not found Jul 14 18:22:15.672768 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:15.672742 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 18:22:15.672837 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:15.672790 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls podName:af335ced-5db5-4190-832e-8ad80e24915b nodeName:}" failed. No retries permitted until 2026-07-14 18:22:16.172775847 +0000 UTC m=+37.710223627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls") pod "cluster-samples-operator-65db858894-bhjwr" (UID: "af335ced-5db5-4190-832e-8ad80e24915b") : secret "samples-operator-tls" not found Jul 14 18:22:15.673787 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.673766 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/42555e5b-8ae2-4677-a74a-bbacb3393030-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-n22g8\" (UID: \"42555e5b-8ae2-4677-a74a-bbacb3393030\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" Jul 14 18:22:15.675085 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.675028 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e2729e0-7391-4488-8c08-285392a648be-cert\") pod \"ingress-canary-hjjpz\" (UID: \"8e2729e0-7391-4488-8c08-285392a648be\") " pod="openshift-ingress-canary/ingress-canary-hjjpz" Jul 14 18:22:15.675264 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.675245 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42555e5b-8ae2-4677-a74a-bbacb3393030-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-n22g8\" (UID: \"42555e5b-8ae2-4677-a74a-bbacb3393030\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" Jul 14 18:22:15.676457 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.676435 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" Jul 14 18:22:15.680653 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.680625 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:15.683141 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.683027 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bqj\" (UniqueName: \"kubernetes.io/projected/8e2729e0-7391-4488-8c08-285392a648be-kube-api-access-d8bqj\") pod \"ingress-canary-hjjpz\" (UID: \"8e2729e0-7391-4488-8c08-285392a648be\") " pod="openshift-ingress-canary/ingress-canary-hjjpz" Jul 14 18:22:15.683903 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.683881 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ppdj\" (UniqueName: \"kubernetes.io/projected/af335ced-5db5-4190-832e-8ad80e24915b-kube-api-access-6ppdj\") pod \"cluster-samples-operator-65db858894-bhjwr\" (UID: \"af335ced-5db5-4190-832e-8ad80e24915b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" Jul 14 18:22:15.685253 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.685227 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krr46\" (UniqueName: \"kubernetes.io/projected/42555e5b-8ae2-4677-a74a-bbacb3393030-kube-api-access-krr46\") pod \"kube-storage-version-migrator-operator-799cb7f958-n22g8\" (UID: \"42555e5b-8ae2-4677-a74a-bbacb3393030\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" Jul 14 18:22:15.686983 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.686939 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9phc\" (UniqueName: \"kubernetes.io/projected/3c17b6fe-41a0-4b30-8dbc-8a9bcf472966-kube-api-access-q9phc\") pod \"network-check-source-869dcfcdcd-jcfxl\" (UID: \"3c17b6fe-41a0-4b30-8dbc-8a9bcf472966\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jcfxl" Jul 14 18:22:15.687924 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.687904 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67d27\" (UniqueName: \"kubernetes.io/projected/24acf903-caba-43ed-bd46-cf61a79524c9-kube-api-access-67d27\") pod \"cluster-monitoring-operator-5f7f494dd4-l4fpb\" (UID: \"24acf903-caba-43ed-bd46-cf61a79524c9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:15.690431 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.690386 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" Jul 14 18:22:15.720115 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.720066 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" Jul 14 18:22:15.742322 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.742043 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jcfxl" Jul 14 18:22:15.773687 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.772279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-config-volume\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:15.773687 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.772574 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:15.773687 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.772665 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2tsq\" (UniqueName: \"kubernetes.io/projected/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-kube-api-access-r2tsq\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:15.773687 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.772720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-tmp-dir\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:15.773687 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.773073 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-tmp-dir\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:15.773687 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:15.773299 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:15.773687 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:15.773368 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls podName:3f139ad7-4fe0-4765-9a55-18231d6dd7cd nodeName:}" failed. No retries permitted until 2026-07-14 18:22:16.27334639 +0000 UTC m=+37.810794168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls") pod "dns-default-7v79r" (UID: "3f139ad7-4fe0-4765-9a55-18231d6dd7cd") : secret "dns-default-metrics-tls" not found Jul 14 18:22:15.773687 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.773642 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-config-volume\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:15.787058 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.786696 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjjpz" Jul 14 18:22:15.822208 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.822078 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:15.822208 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.822144 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2tsq\" (UniqueName: \"kubernetes.io/projected/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-kube-api-access-r2tsq\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:15.874536 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.874407 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-phr5m"] Jul 14 18:22:15.945463 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:15.944785 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-mwmlt"] Jul 14 18:22:15.950702 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:15.950026 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159e6d97_7ed3_42d8_b502_9add9b40eccb.slice/crio-ab3ec24fe268de6fd7bb178d8adbb9b4abd28a7b669f47cf9b5500d05fe89d23 WatchSource:0}: Error finding container ab3ec24fe268de6fd7bb178d8adbb9b4abd28a7b669f47cf9b5500d05fe89d23: Status 404 returned error can't find the container with id ab3ec24fe268de6fd7bb178d8adbb9b4abd28a7b669f47cf9b5500d05fe89d23 Jul 14 18:22:16.004358 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.004268 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg"] Jul 14 18:22:16.008937 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:16.008904 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113b1fbe_22b0_4d6f_8f4f_ede50c3b9242.slice/crio-d82e09612ae019e7264b032594560b666f00e46e89efbc0f2d886dd14c801d9c WatchSource:0}: Error finding container d82e09612ae019e7264b032594560b666f00e46e89efbc0f2d886dd14c801d9c: Status 404 returned error can't find the container with id d82e09612ae019e7264b032594560b666f00e46e89efbc0f2d886dd14c801d9c Jul 14 18:22:16.017939 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.017873 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-tkrhd"] Jul 14 18:22:16.021592 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.021164 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8"] Jul 14 18:22:16.022276 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.022206 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-jcfxl"] Jul 14 18:22:16.022745 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:16.022616 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6392abcd_2ccd_410b_8c10_08a58197237e.slice/crio-c6a29935ac022f581b7cb2367e714bb37b2023887d50554ac05fc30d8f28c751 WatchSource:0}: Error finding container c6a29935ac022f581b7cb2367e714bb37b2023887d50554ac05fc30d8f28c751: Status 404 returned error can't find the container with id c6a29935ac022f581b7cb2367e714bb37b2023887d50554ac05fc30d8f28c751 Jul 14 18:22:16.025040 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:16.025003 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c17b6fe_41a0_4b30_8dbc_8a9bcf472966.slice/crio-cc7311077e1d37a166c40b769d57eeef5facd2249f9e75f13d085292b0fb9a03 WatchSource:0}: Error finding container cc7311077e1d37a166c40b769d57eeef5facd2249f9e75f13d085292b0fb9a03: Status 404 returned error can't find the container with id cc7311077e1d37a166c40b769d57eeef5facd2249f9e75f13d085292b0fb9a03 Jul 14 18:22:16.027252 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:16.027230 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42555e5b_8ae2_4677_a74a_bbacb3393030.slice/crio-c0855d87bb71148d5621c5afffb613794af728d08a28e5dbb23c4baedd17c794 WatchSource:0}: Error finding container c0855d87bb71148d5621c5afffb613794af728d08a28e5dbb23c4baedd17c794: Status 404 returned error can't find the container with id c0855d87bb71148d5621c5afffb613794af728d08a28e5dbb23c4baedd17c794 Jul 14 18:22:16.043857 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.043836 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjjpz"] Jul 14 18:22:16.047380 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:16.047350 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2729e0_7391_4488_8c08_285392a648be.slice/crio-ab0384dac85003e23cbe3d22c96b3df5c1c68c7546d798724d7221b888169303 WatchSource:0}: Error finding container ab0384dac85003e23cbe3d22c96b3df5c1c68c7546d798724d7221b888169303: Status 404 returned error can't find the container with id ab0384dac85003e23cbe3d22c96b3df5c1c68c7546d798724d7221b888169303 Jul 14 18:22:16.077475 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.077449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:16.077610 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.077511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:16.077610 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.077541 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:16.077610 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:16.077563 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:22:16.077610 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:16.077582 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68877f6bf8-lg2z8: secret "image-registry-tls" not found Jul 14 18:22:16.077846 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:16.077660 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 18:22:16.077846 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:16.077657 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls podName:4e730311-d011-495f-a4c7-a0097413011c nodeName:}" failed. No retries permitted until 2026-07-14 18:22:17.077637459 +0000 UTC m=+38.615085231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls") pod "image-registry-68877f6bf8-lg2z8" (UID: "4e730311-d011-495f-a4c7-a0097413011c") : secret "image-registry-tls" not found Jul 14 18:22:16.077846 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:16.077717 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle podName:c7cf164f-807a-4191-acdd-e7d5f3c6de75 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:17.077695718 +0000 UTC m=+38.615143502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle") pod "router-default-59f484b746-8hs6t" (UID: "c7cf164f-807a-4191-acdd-e7d5f3c6de75") : configmap references non-existent config key: service-ca.crt Jul 14 18:22:16.077846 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:16.077742 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs podName:c7cf164f-807a-4191-acdd-e7d5f3c6de75 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:17.077731777 +0000 UTC m=+38.615179542 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs") pod "router-default-59f484b746-8hs6t" (UID: "c7cf164f-807a-4191-acdd-e7d5f3c6de75") : secret "router-metrics-certs-default" not found Jul 14 18:22:16.177992 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.177913 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bhjwr\" (UID: \"af335ced-5db5-4190-832e-8ad80e24915b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" Jul 14 18:22:16.178132 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.178037 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-l4fpb\" (UID: \"24acf903-caba-43ed-bd46-cf61a79524c9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:16.178132 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:16.178053 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 18:22:16.178132 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:16.178120 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls podName:af335ced-5db5-4190-832e-8ad80e24915b nodeName:}" failed. No retries permitted until 2026-07-14 18:22:17.178099489 +0000 UTC m=+38.715547273 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls") pod "cluster-samples-operator-65db858894-bhjwr" (UID: "af335ced-5db5-4190-832e-8ad80e24915b") : secret "samples-operator-tls" not found Jul 14 18:22:16.178266 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:16.178146 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 18:22:16.178266 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:16.178182 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls podName:24acf903-caba-43ed-bd46-cf61a79524c9 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:17.178171385 +0000 UTC m=+38.715619148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-l4fpb" (UID: "24acf903-caba-43ed-bd46-cf61a79524c9") : secret "cluster-monitoring-operator-tls" not found Jul 14 18:22:16.215683 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.215651 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:22:16.215683 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.215679 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:22:16.215924 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.215679 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:22:16.219504 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.219484 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hzvpc\"" Jul 14 18:22:16.219760 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.219738 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 18:22:16.219875 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.219847 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lzfqw\"" Jul 14 18:22:16.219875 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.219866 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 18:22:16.279169 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.279141 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:16.279787 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:16.279257 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:16.279787 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:16.279336 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls podName:3f139ad7-4fe0-4765-9a55-18231d6dd7cd nodeName:}" failed. No retries permitted until 2026-07-14 18:22:17.279321116 +0000 UTC m=+38.816768879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls") pod "dns-default-7v79r" (UID: "3f139ad7-4fe0-4765-9a55-18231d6dd7cd") : secret "dns-default-metrics-tls" not found Jul 14 18:22:16.343818 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.343788 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:16.358438 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.358399 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" event={"ID":"113b1fbe-22b0-4d6f-8f4f-ede50c3b9242","Type":"ContainerStarted","Data":"d82e09612ae019e7264b032594560b666f00e46e89efbc0f2d886dd14c801d9c"} Jul 14 18:22:16.359614 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.359587 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mwmlt" event={"ID":"159e6d97-7ed3-42d8-b502-9add9b40eccb","Type":"ContainerStarted","Data":"ab3ec24fe268de6fd7bb178d8adbb9b4abd28a7b669f47cf9b5500d05fe89d23"} Jul 14 18:22:16.360845 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.360819 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-phr5m" event={"ID":"d1f7e1f6-6948-4d4d-be15-eca40f74051f","Type":"ContainerStarted","Data":"a047a74287d3dfcaafce76693b6db41c1a33dc21792867ff38aef3c05c2a2691"} Jul 14 18:22:16.362047 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.362012 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" event={"ID":"6392abcd-2ccd-410b-8c10-08a58197237e","Type":"ContainerStarted","Data":"c6a29935ac022f581b7cb2367e714bb37b2023887d50554ac05fc30d8f28c751"} Jul 14 18:22:16.363280 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.363255 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjjpz" event={"ID":"8e2729e0-7391-4488-8c08-285392a648be","Type":"ContainerStarted","Data":"ab0384dac85003e23cbe3d22c96b3df5c1c68c7546d798724d7221b888169303"} Jul 14 18:22:16.364344 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.364317 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jcfxl" event={"ID":"3c17b6fe-41a0-4b30-8dbc-8a9bcf472966","Type":"ContainerStarted","Data":"cc7311077e1d37a166c40b769d57eeef5facd2249f9e75f13d085292b0fb9a03"} Jul 14 18:22:16.365430 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:16.365410 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" event={"ID":"42555e5b-8ae2-4677-a74a-bbacb3393030","Type":"ContainerStarted","Data":"c0855d87bb71148d5621c5afffb613794af728d08a28e5dbb23c4baedd17c794"} Jul 14 18:22:17.087662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:17.087623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:17.087830 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:17.087766 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:17.087830 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:17.087781 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:22:17.087830 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:17.087800 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:17.088001 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:17.087803 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68877f6bf8-lg2z8: secret "image-registry-tls" not found Jul 14 18:22:17.088001 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:17.087906 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle podName:c7cf164f-807a-4191-acdd-e7d5f3c6de75 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:19.087886346 +0000 UTC m=+40.625334122 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle") pod "router-default-59f484b746-8hs6t" (UID: "c7cf164f-807a-4191-acdd-e7d5f3c6de75") : configmap references non-existent config key: service-ca.crt Jul 14 18:22:17.088001 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:17.087947 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls podName:4e730311-d011-495f-a4c7-a0097413011c nodeName:}" failed. No retries permitted until 2026-07-14 18:22:19.087928131 +0000 UTC m=+40.625375902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls") pod "image-registry-68877f6bf8-lg2z8" (UID: "4e730311-d011-495f-a4c7-a0097413011c") : secret "image-registry-tls" not found Jul 14 18:22:17.088147 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:17.088063 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 18:22:17.088147 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:17.088099 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs podName:c7cf164f-807a-4191-acdd-e7d5f3c6de75 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:19.088087329 +0000 UTC m=+40.625535094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs") pod "router-default-59f484b746-8hs6t" (UID: "c7cf164f-807a-4191-acdd-e7d5f3c6de75") : secret "router-metrics-certs-default" not found Jul 14 18:22:17.188705 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:17.188662 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-l4fpb\" (UID: \"24acf903-caba-43ed-bd46-cf61a79524c9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:17.188868 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:17.188777 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bhjwr\" (UID: \"af335ced-5db5-4190-832e-8ad80e24915b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" Jul 14 18:22:17.188996 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:17.188981 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 18:22:17.189066 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:17.189061 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls podName:af335ced-5db5-4190-832e-8ad80e24915b nodeName:}" failed. No retries permitted until 2026-07-14 18:22:19.189041419 +0000 UTC m=+40.726489185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls") pod "cluster-samples-operator-65db858894-bhjwr" (UID: "af335ced-5db5-4190-832e-8ad80e24915b") : secret "samples-operator-tls" not found Jul 14 18:22:17.189573 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:17.189471 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 18:22:17.189573 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:17.189525 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls podName:24acf903-caba-43ed-bd46-cf61a79524c9 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:19.189508552 +0000 UTC m=+40.726956320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-l4fpb" (UID: "24acf903-caba-43ed-bd46-cf61a79524c9") : secret "cluster-monitoring-operator-tls" not found Jul 14 18:22:17.290240 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:17.290023 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:17.290240 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:17.290159 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:17.290240 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:17.290237 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls podName:3f139ad7-4fe0-4765-9a55-18231d6dd7cd nodeName:}" failed. No retries permitted until 2026-07-14 18:22:19.290214711 +0000 UTC m=+40.827662477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls") pod "dns-default-7v79r" (UID: "3f139ad7-4fe0-4765-9a55-18231d6dd7cd") : secret "dns-default-metrics-tls" not found Jul 14 18:22:19.109895 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:19.109857 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:19.110356 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:19.109915 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:19.110356 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:19.109940 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:19.110356 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:19.110021 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:22:19.110356 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:19.110043 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68877f6bf8-lg2z8: secret "image-registry-tls" not found Jul 14 18:22:19.110356 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:19.110064 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 18:22:19.110356 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:19.110087 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle podName:c7cf164f-807a-4191-acdd-e7d5f3c6de75 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:23.110068935 +0000 UTC m=+44.647516718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle") pod "router-default-59f484b746-8hs6t" (UID: "c7cf164f-807a-4191-acdd-e7d5f3c6de75") : configmap references non-existent config key: service-ca.crt Jul 14 18:22:19.110356 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:19.110118 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs podName:c7cf164f-807a-4191-acdd-e7d5f3c6de75 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:23.11009833 +0000 UTC m=+44.647546122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs") pod "router-default-59f484b746-8hs6t" (UID: "c7cf164f-807a-4191-acdd-e7d5f3c6de75") : secret "router-metrics-certs-default" not found Jul 14 18:22:19.110356 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:19.110144 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls podName:4e730311-d011-495f-a4c7-a0097413011c nodeName:}" failed. No retries permitted until 2026-07-14 18:22:23.110126997 +0000 UTC m=+44.647574772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls") pod "image-registry-68877f6bf8-lg2z8" (UID: "4e730311-d011-495f-a4c7-a0097413011c") : secret "image-registry-tls" not found Jul 14 18:22:19.211053 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:19.211016 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-l4fpb\" (UID: \"24acf903-caba-43ed-bd46-cf61a79524c9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:19.211222 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:19.211111 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bhjwr\" (UID: \"af335ced-5db5-4190-832e-8ad80e24915b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" Jul 14 18:22:19.211222 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:19.211203 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 18:22:19.211355 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:19.211238 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 18:22:19.211355 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:19.211299 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls podName:24acf903-caba-43ed-bd46-cf61a79524c9 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:23.211264082 +0000 UTC m=+44.748711851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-l4fpb" (UID: "24acf903-caba-43ed-bd46-cf61a79524c9") : secret "cluster-monitoring-operator-tls" not found Jul 14 18:22:19.211355 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:19.211320 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls podName:af335ced-5db5-4190-832e-8ad80e24915b nodeName:}" failed. No retries permitted until 2026-07-14 18:22:23.211310072 +0000 UTC m=+44.748757839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls") pod "cluster-samples-operator-65db858894-bhjwr" (UID: "af335ced-5db5-4190-832e-8ad80e24915b") : secret "samples-operator-tls" not found Jul 14 18:22:19.311986 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:19.311951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:19.312154 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:19.312124 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:19.312199 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:19.312192 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls podName:3f139ad7-4fe0-4765-9a55-18231d6dd7cd nodeName:}" failed. No retries permitted until 2026-07-14 18:22:23.312175716 +0000 UTC m=+44.849623479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls") pod "dns-default-7v79r" (UID: "3f139ad7-4fe0-4765-9a55-18231d6dd7cd") : secret "dns-default-metrics-tls" not found Jul 14 18:22:20.549185 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:20.548914 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:20.695302 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:20.695246 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:21.842126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:21.842090 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:22.191791 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:22.191754 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:23.144386 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:23.144341 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:23.144386 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:23.144387 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:23.144793 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:23.144488 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:23.144793 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:23.144553 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle podName:c7cf164f-807a-4191-acdd-e7d5f3c6de75 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:31.144527904 +0000 UTC m=+52.681975685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle") pod "router-default-59f484b746-8hs6t" (UID: "c7cf164f-807a-4191-acdd-e7d5f3c6de75") : configmap references non-existent config key: service-ca.crt Jul 14 18:22:23.144793 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:23.144578 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:22:23.144793 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:23.144580 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 18:22:23.144793 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:23.144592 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68877f6bf8-lg2z8: secret "image-registry-tls" not found Jul 14 18:22:23.144793 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:23.144641 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs podName:c7cf164f-807a-4191-acdd-e7d5f3c6de75 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:31.144626798 +0000 UTC m=+52.682074561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs") pod "router-default-59f484b746-8hs6t" (UID: "c7cf164f-807a-4191-acdd-e7d5f3c6de75") : secret "router-metrics-certs-default" not found Jul 14 18:22:23.144793 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:23.144657 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls podName:4e730311-d011-495f-a4c7-a0097413011c nodeName:}" failed. No retries permitted until 2026-07-14 18:22:31.144650794 +0000 UTC m=+52.682098556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls") pod "image-registry-68877f6bf8-lg2z8" (UID: "4e730311-d011-495f-a4c7-a0097413011c") : secret "image-registry-tls" not found Jul 14 18:22:23.245346 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:23.245319 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-l4fpb\" (UID: \"24acf903-caba-43ed-bd46-cf61a79524c9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:23.245508 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:23.245462 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 18:22:23.245508 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:23.245486 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bhjwr\" (UID: \"af335ced-5db5-4190-832e-8ad80e24915b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" Jul 14 18:22:23.245583 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:23.245567 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls podName:24acf903-caba-43ed-bd46-cf61a79524c9 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:31.245545473 +0000 UTC m=+52.782993236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-l4fpb" (UID: "24acf903-caba-43ed-bd46-cf61a79524c9") : secret "cluster-monitoring-operator-tls" not found Jul 14 18:22:23.245626 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:23.245606 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 18:22:23.245664 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:23.245647 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls podName:af335ced-5db5-4190-832e-8ad80e24915b nodeName:}" failed. No retries permitted until 2026-07-14 18:22:31.245634233 +0000 UTC m=+52.783081995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls") pod "cluster-samples-operator-65db858894-bhjwr" (UID: "af335ced-5db5-4190-832e-8ad80e24915b") : secret "samples-operator-tls" not found Jul 14 18:22:23.346564 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:23.346523 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:23.346725 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:23.346687 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:23.346783 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:23.346764 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls podName:3f139ad7-4fe0-4765-9a55-18231d6dd7cd nodeName:}" failed. No retries permitted until 2026-07-14 18:22:31.346742006 +0000 UTC m=+52.884189779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls") pod "dns-default-7v79r" (UID: "3f139ad7-4fe0-4765-9a55-18231d6dd7cd") : secret "dns-default-metrics-tls" not found Jul 14 18:22:23.721352 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:23.721320 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:23.836094 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:23.836061 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:26.399469 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.399385 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjjpz" event={"ID":"8e2729e0-7391-4488-8c08-285392a648be","Type":"ContainerStarted","Data":"206521ad965168f8f716e217935d9bf55a4c86bdeea9ddf40e02464dd33f8d67"} Jul 14 18:22:26.401112 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.401059 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jcfxl" event={"ID":"3c17b6fe-41a0-4b30-8dbc-8a9bcf472966","Type":"ContainerStarted","Data":"5634ed3da999557cc68eafc661a51bdba1c92cf01749e107ad38a7cfe29c2c01"} Jul 14 18:22:26.402833 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.402787 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" event={"ID":"42555e5b-8ae2-4677-a74a-bbacb3393030","Type":"ContainerStarted","Data":"c3c563e837e6f31646c6f91c8e2634ebae33ac15e51ed15d7c38b9441b399f05"} Jul 14 18:22:26.404462 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.404420 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" event={"ID":"113b1fbe-22b0-4d6f-8f4f-ede50c3b9242","Type":"ContainerStarted","Data":"4e52a1b71cc3d2f2866eddb4aaa79752ba18b51f3862dba3ec83e6a0210bcf69"} Jul 14 18:22:26.406196 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.406180 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/0.log" Jul 14 18:22:26.406316 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.406212 2573 generic.go:356] "Generic (PLEG): container finished" podID="159e6d97-7ed3-42d8-b502-9add9b40eccb" containerID="e4ba0c8dbc09fa51c389164020626018b52fba912de82010710f547b86bd421d" exitCode=255 Jul 14 18:22:26.406316 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.406278 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mwmlt" event={"ID":"159e6d97-7ed3-42d8-b502-9add9b40eccb","Type":"ContainerDied","Data":"e4ba0c8dbc09fa51c389164020626018b52fba912de82010710f547b86bd421d"} Jul 14 18:22:26.406501 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.406477 2573 scope.go:117] "RemoveContainer" containerID="e4ba0c8dbc09fa51c389164020626018b52fba912de82010710f547b86bd421d" Jul 14 18:22:26.408008 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.407970 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-phr5m" event={"ID":"d1f7e1f6-6948-4d4d-be15-eca40f74051f","Type":"ContainerStarted","Data":"22fe929f3ecd20916b94b336d5b3f7b26f04665c0a377ac0eb89aaf5c793be7e"} Jul 14 18:22:26.409747 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.409707 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" event={"ID":"6392abcd-2ccd-410b-8c10-08a58197237e","Type":"ContainerStarted","Data":"d50c5b15c0501e9da6670186c91b502b55545c041bfbe71a69976685aaf42431"} Jul 14 18:22:26.412732 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.412711 2573 generic.go:356] "Generic (PLEG): container finished" podID="9ad7d617-55f3-43d9-8682-daf673a06f1c" containerID="ca93ecbc797daf1b1635007f3134038d2039c8b8dc0e764d16842f3ffdc98487" exitCode=0 Jul 14 18:22:26.412829 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.412741 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpz5d" event={"ID":"9ad7d617-55f3-43d9-8682-daf673a06f1c","Type":"ContainerDied","Data":"ca93ecbc797daf1b1635007f3134038d2039c8b8dc0e764d16842f3ffdc98487"} Jul 14 18:22:26.425638 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.425598 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hjjpz" podStartSLOduration=1.7289641279999999 podStartE2EDuration="11.425585495s" totalImagesPullingTime="9.696621367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:26.424625785 +0000 UTC m=+47.962073569" watchObservedRunningTime="2026-07-14 18:22:26.425585495 +0000 UTC m=+47.963033257" Jul 14 18:22:26.449392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.448571 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-phr5m" podStartSLOduration=15.605762755 podStartE2EDuration="25.448551596s" totalImagesPullingTime="9.842788841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:26.447021385 +0000 UTC m=+47.984469171" watchObservedRunningTime="2026-07-14 18:22:26.448551596 +0000 UTC m=+47.985999383" Jul 14 18:22:26.473511 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.473460 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" podStartSLOduration=15.699752945 podStartE2EDuration="25.473440261s" totalImagesPullingTime="9.773687316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:26.470925479 +0000 UTC m=+48.008373264" watchObservedRunningTime="2026-07-14 18:22:26.473440261 +0000 UTC m=+48.010888048" Jul 14 18:22:26.545570 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.545517 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" podStartSLOduration=14.830446288 podStartE2EDuration="24.545494942s" totalImagesPullingTime="9.715048654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:26.543854355 +0000 UTC m=+48.081302137" watchObservedRunningTime="2026-07-14 18:22:26.545494942 +0000 UTC m=+48.082942727" Jul 14 18:22:26.569940 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.569648 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:26.610222 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.610169 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" podStartSLOduration=14.889317566 podStartE2EDuration="24.610146498s" totalImagesPullingTime="9.720828932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:26.608305222 +0000 UTC m=+48.145753003" watchObservedRunningTime="2026-07-14 18:22:26.610146498 +0000 UTC m=+48.147594285" Jul 14 18:22:26.633739 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.633544 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jcfxl" podStartSLOduration=15.918376861 podStartE2EDuration="25.633524349s" totalImagesPullingTime="9.715147488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:26.630436783 +0000 UTC m=+48.167884571" watchObservedRunningTime="2026-07-14 18:22:26.633524349 +0000 UTC m=+48.170972135" Jul 14 18:22:26.672625 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:26.672592 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:27.061602 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.061569 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-znj4p"] Jul 14 18:22:27.080889 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.080865 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.088268 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.088103 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 18:22:27.089399 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.089312 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 18:22:27.089399 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.089339 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vtlhx\"" Jul 14 18:22:27.090165 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.090140 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-znj4p"] Jul 14 18:22:27.127043 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.127013 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:27.184846 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.184810 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8hwv\" (UniqueName: \"kubernetes.io/projected/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-api-access-k8hwv\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.185028 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.184958 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1091f487-d0e7-4d2d-96ac-1849adce0fa6-data-volume\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.185028 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.184984 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.185028 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.185004 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.185175 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.185063 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1091f487-d0e7-4d2d-96ac-1849adce0fa6-crio-socket\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.217654 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.217618 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:27.286090 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.286053 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8hwv\" (UniqueName: \"kubernetes.io/projected/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-api-access-k8hwv\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.286270 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.286180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1091f487-d0e7-4d2d-96ac-1849adce0fa6-data-volume\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.286270 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.286209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.286270 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.286233 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.286517 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:27.286340 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:22:27.286517 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.286384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1091f487-d0e7-4d2d-96ac-1849adce0fa6-crio-socket\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.286517 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:27.286412 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-rbac-proxy-cm podName:1091f487-d0e7-4d2d-96ac-1849adce0fa6 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:27.786391369 +0000 UTC m=+49.323839145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-znj4p" (UID: "1091f487-d0e7-4d2d-96ac-1849adce0fa6") : configmap "kube-rbac-proxy" not found Jul 14 18:22:27.286517 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:27.286348 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:22:27.286517 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:27.286493 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls podName:1091f487-d0e7-4d2d-96ac-1849adce0fa6 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:27.78647424 +0000 UTC m=+49.323922009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-znj4p" (UID: "1091f487-d0e7-4d2d-96ac-1849adce0fa6") : secret "insights-runtime-extractor-tls" not found Jul 14 18:22:27.286734 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.286579 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1091f487-d0e7-4d2d-96ac-1849adce0fa6-crio-socket\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.286734 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.286596 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1091f487-d0e7-4d2d-96ac-1849adce0fa6-data-volume\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.299469 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.299440 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8hwv\" (UniqueName: \"kubernetes.io/projected/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-api-access-k8hwv\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.419520 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.419439 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:22:27.419968 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.419949 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/0.log" Jul 14 18:22:27.420011 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.419993 2573 generic.go:356] "Generic (PLEG): container finished" podID="159e6d97-7ed3-42d8-b502-9add9b40eccb" containerID="7b1e154b809a1bf0c6d271a5980a6c5a3238de5c628fae884815a134c18c38ca" exitCode=255 Jul 14 18:22:27.420128 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.420082 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mwmlt" event={"ID":"159e6d97-7ed3-42d8-b502-9add9b40eccb","Type":"ContainerDied","Data":"7b1e154b809a1bf0c6d271a5980a6c5a3238de5c628fae884815a134c18c38ca"} Jul 14 18:22:27.420207 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.420153 2573 scope.go:117] "RemoveContainer" containerID="e4ba0c8dbc09fa51c389164020626018b52fba912de82010710f547b86bd421d" Jul 14 18:22:27.420394 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.420376 2573 scope.go:117] "RemoveContainer" containerID="7b1e154b809a1bf0c6d271a5980a6c5a3238de5c628fae884815a134c18c38ca" Jul 14 18:22:27.420763 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:27.420736 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-mwmlt_openshift-console-operator(159e6d97-7ed3-42d8-b502-9add9b40eccb)\"" pod="openshift-console-operator/console-operator-7866497494-mwmlt" podUID="159e6d97-7ed3-42d8-b502-9add9b40eccb" Jul 14 18:22:27.424107 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.424086 2573 generic.go:356] "Generic (PLEG): container finished" podID="9ad7d617-55f3-43d9-8682-daf673a06f1c" containerID="e81a29a6f3d09814ca36a3c39072020c49e2d2201b43f595ebaf3eee8bd54a12" exitCode=0 Jul 14 18:22:27.424243 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.424213 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpz5d" event={"ID":"9ad7d617-55f3-43d9-8682-daf673a06f1c","Type":"ContainerDied","Data":"e81a29a6f3d09814ca36a3c39072020c49e2d2201b43f595ebaf3eee8bd54a12"} Jul 14 18:22:27.761784 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.761736 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:27.795800 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.795760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.795800 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:27.795804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:27.796035 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:27.795895 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:22:27.796035 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:27.795991 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-rbac-proxy-cm podName:1091f487-d0e7-4d2d-96ac-1849adce0fa6 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:28.795971562 +0000 UTC m=+50.333419335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-znj4p" (UID: "1091f487-d0e7-4d2d-96ac-1849adce0fa6") : configmap "kube-rbac-proxy" not found Jul 14 18:22:27.796035 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:27.796005 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:22:27.796201 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:27.796073 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls podName:1091f487-d0e7-4d2d-96ac-1849adce0fa6 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:28.796054637 +0000 UTC m=+50.333502418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-znj4p" (UID: "1091f487-d0e7-4d2d-96ac-1849adce0fa6") : secret "insights-runtime-extractor-tls" not found Jul 14 18:22:28.429279 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:28.429251 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:22:28.429722 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:28.429684 2573 scope.go:117] "RemoveContainer" containerID="7b1e154b809a1bf0c6d271a5980a6c5a3238de5c628fae884815a134c18c38ca" Jul 14 18:22:28.429927 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:28.429894 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-mwmlt_openshift-console-operator(159e6d97-7ed3-42d8-b502-9add9b40eccb)\"" pod="openshift-console-operator/console-operator-7866497494-mwmlt" podUID="159e6d97-7ed3-42d8-b502-9add9b40eccb" Jul 14 18:22:28.432605 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:28.432582 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wpz5d" event={"ID":"9ad7d617-55f3-43d9-8682-daf673a06f1c","Type":"ContainerStarted","Data":"1d4874116058e6039bf049cb0aa2e6b94bedc49f808b4506b42efe3c2f6e43a1"} Jul 14 18:22:28.490266 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:28.490210 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wpz5d" podStartSLOduration=9.15673179 podStartE2EDuration="49.490194381s" totalImagesPullingTime="40.333462591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:28.489610296 +0000 UTC m=+50.027058081" watchObservedRunningTime="2026-07-14 18:22:28.490194381 +0000 UTC m=+50.027642171" Jul 14 18:22:28.805126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:28.805088 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:28.805126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:28.805128 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:28.805388 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:28.805221 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:22:28.805388 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:28.805299 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-rbac-proxy-cm podName:1091f487-d0e7-4d2d-96ac-1849adce0fa6 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:30.805267256 +0000 UTC m=+52.342715023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-znj4p" (UID: "1091f487-d0e7-4d2d-96ac-1849adce0fa6") : configmap "kube-rbac-proxy" not found Jul 14 18:22:28.805388 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:28.805226 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:22:28.805388 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:28.805349 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls podName:1091f487-d0e7-4d2d-96ac-1849adce0fa6 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:30.805337824 +0000 UTC m=+52.342785587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-znj4p" (UID: "1091f487-d0e7-4d2d-96ac-1849adce0fa6") : secret "insights-runtime-extractor-tls" not found Jul 14 18:22:29.251117 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:29.251082 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tjpz8_5bdb299c-c772-4af9-8286-6fb4c2371885/dns-node-resolver/0.log" Jul 14 18:22:30.051725 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:30.051698 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kjlrj_80ba9b7f-15c6-4fc0-82ab-8286c596aff1/node-ca/0.log" Jul 14 18:22:30.486595 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:30.486554 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:30.568129 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:30.568096 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:30.651597 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:30.651569 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjjpz_8e2729e0-7391-4488-8c08-285392a648be/serve-healthcheck-canary/0.log" Jul 14 18:22:30.791701 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:30.791600 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:30.824438 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:30.824405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:30.824438 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:30.824440 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:30.824681 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:30.824565 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:22:30.824681 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:30.824628 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls podName:1091f487-d0e7-4d2d-96ac-1849adce0fa6 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:34.824608785 +0000 UTC m=+56.362056566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-znj4p" (UID: "1091f487-d0e7-4d2d-96ac-1849adce0fa6") : secret "insights-runtime-extractor-tls" not found Jul 14 18:22:30.824681 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:30.824653 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:22:30.824836 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:30.824725 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-rbac-proxy-cm podName:1091f487-d0e7-4d2d-96ac-1849adce0fa6 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:34.824711569 +0000 UTC m=+56.362159332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-znj4p" (UID: "1091f487-d0e7-4d2d-96ac-1849adce0fa6") : configmap "kube-rbac-proxy" not found Jul 14 18:22:31.149215 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:31.149107 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:31.227893 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:31.227864 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:31.228084 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:31.227907 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:31.228084 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:31.227926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:31.228203 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:31.228083 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 18:22:31.228203 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:31.228083 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle podName:c7cf164f-807a-4191-acdd-e7d5f3c6de75 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:47.228060127 +0000 UTC m=+68.765507902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle") pod "router-default-59f484b746-8hs6t" (UID: "c7cf164f-807a-4191-acdd-e7d5f3c6de75") : configmap references non-existent config key: service-ca.crt Jul 14 18:22:31.228203 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:31.228136 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs podName:c7cf164f-807a-4191-acdd-e7d5f3c6de75 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:47.228124601 +0000 UTC m=+68.765572364 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs") pod "router-default-59f484b746-8hs6t" (UID: "c7cf164f-807a-4191-acdd-e7d5f3c6de75") : secret "router-metrics-certs-default" not found Jul 14 18:22:31.228203 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:31.228087 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:22:31.228203 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:31.228145 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68877f6bf8-lg2z8: secret "image-registry-tls" not found Jul 14 18:22:31.228203 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:31.228170 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls podName:4e730311-d011-495f-a4c7-a0097413011c nodeName:}" failed. No retries permitted until 2026-07-14 18:22:47.228162395 +0000 UTC m=+68.765610161 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls") pod "image-registry-68877f6bf8-lg2z8" (UID: "4e730311-d011-495f-a4c7-a0097413011c") : secret "image-registry-tls" not found Jul 14 18:22:31.323256 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:31.323219 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:31.329399 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:31.329374 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-l4fpb\" (UID: \"24acf903-caba-43ed-bd46-cf61a79524c9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:31.329521 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:31.329463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bhjwr\" (UID: \"af335ced-5db5-4190-832e-8ad80e24915b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" Jul 14 18:22:31.329567 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:31.329516 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 18:22:31.329606 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:31.329595 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls podName:24acf903-caba-43ed-bd46-cf61a79524c9 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:47.329578482 +0000 UTC m=+68.867026254 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-l4fpb" (UID: "24acf903-caba-43ed-bd46-cf61a79524c9") : secret "cluster-monitoring-operator-tls" not found Jul 14 18:22:31.329606 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:31.329597 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 18:22:31.329677 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:31.329626 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls podName:af335ced-5db5-4190-832e-8ad80e24915b nodeName:}" failed. No retries permitted until 2026-07-14 18:22:47.329619626 +0000 UTC m=+68.867067394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls") pod "cluster-samples-operator-65db858894-bhjwr" (UID: "af335ced-5db5-4190-832e-8ad80e24915b") : secret "samples-operator-tls" not found Jul 14 18:22:31.430264 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:31.430184 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:31.430430 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:31.430353 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:31.430430 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:31.430426 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls podName:3f139ad7-4fe0-4765-9a55-18231d6dd7cd nodeName:}" failed. No retries permitted until 2026-07-14 18:22:47.430407766 +0000 UTC m=+68.967855530 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls") pod "dns-default-7v79r" (UID: "3f139ad7-4fe0-4765-9a55-18231d6dd7cd") : secret "dns-default-metrics-tls" not found Jul 14 18:22:34.688936 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:34.688897 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:34.786798 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:34.786752 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:34.859368 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:34.859334 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:34.859553 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:34.859376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:34.859620 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:34.859600 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:22:34.859693 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:34.859681 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls podName:1091f487-d0e7-4d2d-96ac-1849adce0fa6 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:42.859660357 +0000 UTC m=+64.397108143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-znj4p" (UID: "1091f487-d0e7-4d2d-96ac-1849adce0fa6") : secret "insights-runtime-extractor-tls" not found Jul 14 18:22:34.859899 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:34.859876 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1091f487-d0e7-4d2d-96ac-1849adce0fa6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:34.910360 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:34.910319 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:35.018165 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:35.018077 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:35.111267 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:35.111233 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:35.639397 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:35.639363 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:35.639556 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:35.639421 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:22:35.639762 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:35.639750 2573 scope.go:117] "RemoveContainer" containerID="7b1e154b809a1bf0c6d271a5980a6c5a3238de5c628fae884815a134c18c38ca" Jul 14 18:22:35.639906 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:22:35.639890 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-mwmlt_openshift-console-operator(159e6d97-7ed3-42d8-b502-9add9b40eccb)\"" pod="openshift-console-operator/console-operator-7866497494-mwmlt" podUID="159e6d97-7ed3-42d8-b502-9add9b40eccb" Jul 14 18:22:38.671351 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:38.671309 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:38.800006 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:38.799972 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:38.998279 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:38.998246 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:39.192093 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:39.192063 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:39.308183 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:39.308081 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:39.396402 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:39.396370 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:40.344742 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:40.344716 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8pxlk" Jul 14 18:22:40.420427 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:40.420389 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:40.615447 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:40.615369 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:40.779147 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:40.779111 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:41.011575 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:41.011540 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:41.185762 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:41.185730 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:41.367015 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:41.366937 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:42.374338 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:42.374303 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:42.757876 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:42.757844 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:42.934018 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:42.933982 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:42.936275 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:42.936250 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1091f487-d0e7-4d2d-96ac-1849adce0fa6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znj4p\" (UID: \"1091f487-d0e7-4d2d-96ac-1849adce0fa6\") " pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:42.957441 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:42.957412 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:42.995157 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:42.995131 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vtlhx\"" Jul 14 18:22:43.002411 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:43.002394 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-znj4p" Jul 14 18:22:43.152277 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:43.152249 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-znj4p"] Jul 14 18:22:43.155582 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:43.155560 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1091f487_d0e7_4d2d_96ac_1849adce0fa6.slice/crio-18bd90b36682a30c898bdbeef78898afb47174febef0cb2356c5db8142551474 WatchSource:0}: Error finding container 18bd90b36682a30c898bdbeef78898afb47174febef0cb2356c5db8142551474: Status 404 returned error can't find the container with id 18bd90b36682a30c898bdbeef78898afb47174febef0cb2356c5db8142551474 Jul 14 18:22:43.191472 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:43.191440 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:43.392183 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:43.392097 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:43.479403 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:43.479369 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-znj4p" event={"ID":"1091f487-d0e7-4d2d-96ac-1849adce0fa6","Type":"ContainerStarted","Data":"18bd90b36682a30c898bdbeef78898afb47174febef0cb2356c5db8142551474"} Jul 14 18:22:43.801252 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:43.801219 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:44.375683 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.375654 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:44.650569 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.650484 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs\") pod \"network-metrics-daemon-29dsq\" (UID: \"2a487018-2ca6-4484-892b-8998a760a384\") " pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:22:44.651003 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.650572 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:22:44.651003 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.650621 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjch\" (UniqueName: \"kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch\") pod \"network-check-target-rdlqb\" (UID: \"b5910206-776a-40cd-92ec-e5a154cc261a\") " pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:22:44.653471 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.653445 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkjch\" (UniqueName: \"kubernetes.io/projected/b5910206-776a-40cd-92ec-e5a154cc261a-kube-api-access-fkjch\") pod \"network-check-target-rdlqb\" (UID: \"b5910206-776a-40cd-92ec-e5a154cc261a\") " pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:22:44.654038 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.654018 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 18:22:44.655244 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.655226 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 18:22:44.663327 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.663302 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a487018-2ca6-4484-892b-8998a760a384-metrics-certs\") pod \"network-metrics-daemon-29dsq\" (UID: \"2a487018-2ca6-4484-892b-8998a760a384\") " pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:22:44.663432 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.663416 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0049cf44-b6fe-4c7c-8e53-adbd70b7eb46-original-pull-secret\") pod \"global-pull-secret-syncer-gm8z4\" (UID: \"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46\") " pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:22:44.734553 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.734524 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hzvpc\"" Jul 14 18:22:44.736644 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.736622 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gm8z4" Jul 14 18:22:44.739861 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.739834 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29dsq" Jul 14 18:22:44.745492 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.745472 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lzfqw\"" Jul 14 18:22:44.753122 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.753102 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:22:44.825224 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.825181 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:44.987853 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:44.987825 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:45.177011 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:45.176968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:45.463300 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:45.463260 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:46.888075 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:46.887994 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gm8z4"] Jul 14 18:22:46.892659 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:46.892630 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0049cf44_b6fe_4c7c_8e53_adbd70b7eb46.slice/crio-c27d6ccca7da2a61e0998b83158d16073f1122c4e112a8ad53be96e8a6a1c122 WatchSource:0}: Error finding container c27d6ccca7da2a61e0998b83158d16073f1122c4e112a8ad53be96e8a6a1c122: Status 404 returned error can't find the container with id c27d6ccca7da2a61e0998b83158d16073f1122c4e112a8ad53be96e8a6a1c122 Jul 14 18:22:47.124591 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.124504 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rdlqb"] Jul 14 18:22:47.127335 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.127308 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29dsq"] Jul 14 18:22:47.127882 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:47.127850 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5910206_776a_40cd_92ec_e5a154cc261a.slice/crio-4f2f241cde8aa4493c8c60cf8dc0cfa5ee5255e8254cb2a43ca2a9f0420494bb WatchSource:0}: Error finding container 4f2f241cde8aa4493c8c60cf8dc0cfa5ee5255e8254cb2a43ca2a9f0420494bb: Status 404 returned error can't find the container with id 4f2f241cde8aa4493c8c60cf8dc0cfa5ee5255e8254cb2a43ca2a9f0420494bb Jul 14 18:22:47.129895 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:47.129871 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a487018_2ca6_4484_892b_8998a760a384.slice/crio-7b333e7b0a88458172af9e4c591f5d3c4612b06f72c9809c80b26dea817f5fa9 WatchSource:0}: Error finding container 7b333e7b0a88458172af9e4c591f5d3c4612b06f72c9809c80b26dea817f5fa9: Status 404 returned error can't find the container with id 7b333e7b0a88458172af9e4c591f5d3c4612b06f72c9809c80b26dea817f5fa9 Jul 14 18:22:47.278165 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.278132 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:47.278165 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.278174 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:47.278393 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.278191 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:47.278792 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.278765 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7cf164f-807a-4191-acdd-e7d5f3c6de75-service-ca-bundle\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:47.280500 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.280473 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls\") pod \"image-registry-68877f6bf8-lg2z8\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:47.280596 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.280531 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7cf164f-807a-4191-acdd-e7d5f3c6de75-metrics-certs\") pod \"router-default-59f484b746-8hs6t\" (UID: \"c7cf164f-807a-4191-acdd-e7d5f3c6de75\") " pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:47.379614 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.379524 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-l4fpb\" (UID: \"24acf903-caba-43ed-bd46-cf61a79524c9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:47.379614 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.379593 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bhjwr\" (UID: \"af335ced-5db5-4190-832e-8ad80e24915b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" Jul 14 18:22:47.382745 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.382678 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/af335ced-5db5-4190-832e-8ad80e24915b-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-bhjwr\" (UID: \"af335ced-5db5-4190-832e-8ad80e24915b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" Jul 14 18:22:47.382922 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.382899 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/24acf903-caba-43ed-bd46-cf61a79524c9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-l4fpb\" (UID: \"24acf903-caba-43ed-bd46-cf61a79524c9\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:47.423952 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.423920 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-psptv\"" Jul 14 18:22:47.430940 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.430919 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:47.466207 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.466181 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xxw7v\"" Jul 14 18:22:47.473079 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.473057 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:47.480632 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.480603 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:47.483233 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.483181 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f139ad7-4fe0-4765-9a55-18231d6dd7cd-metrics-tls\") pod \"dns-default-7v79r\" (UID: \"3f139ad7-4fe0-4765-9a55-18231d6dd7cd\") " pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:47.499156 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.499023 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-znj4p" event={"ID":"1091f487-d0e7-4d2d-96ac-1849adce0fa6","Type":"ContainerStarted","Data":"3e23982697cae6ab9d247aa1a5e269a474a3424cd967055c53e535bda94309f7"} Jul 14 18:22:47.499156 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.499095 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-znj4p" event={"ID":"1091f487-d0e7-4d2d-96ac-1849adce0fa6","Type":"ContainerStarted","Data":"7166d9e421d9dc1479bc53ab58a145a4b31735e95d6e4ba78e7ae4762e7b241d"} Jul 14 18:22:47.502616 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.502371 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rdlqb" event={"ID":"b5910206-776a-40cd-92ec-e5a154cc261a","Type":"ContainerStarted","Data":"37ba561cb2400faa0fd256f283c8ee07cb46afee4597a8e0884df15b63d29550"} Jul 14 18:22:47.502616 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.502419 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rdlqb" event={"ID":"b5910206-776a-40cd-92ec-e5a154cc261a","Type":"ContainerStarted","Data":"4f2f241cde8aa4493c8c60cf8dc0cfa5ee5255e8254cb2a43ca2a9f0420494bb"} Jul 14 18:22:47.502616 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.502500 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:22:47.504635 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.504605 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gm8z4" event={"ID":"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46","Type":"ContainerStarted","Data":"c27d6ccca7da2a61e0998b83158d16073f1122c4e112a8ad53be96e8a6a1c122"} Jul 14 18:22:47.506838 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.506816 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29dsq" event={"ID":"2a487018-2ca6-4484-892b-8998a760a384","Type":"ContainerStarted","Data":"7b333e7b0a88458172af9e4c591f5d3c4612b06f72c9809c80b26dea817f5fa9"} Jul 14 18:22:47.511020 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.510612 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dz5cv\"" Jul 14 18:22:47.518344 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.518316 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" Jul 14 18:22:47.529671 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.527910 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rdlqb" podStartSLOduration=68.527880848 podStartE2EDuration="1m8.527880848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:47.526923773 +0000 UTC m=+69.064371558" watchObservedRunningTime="2026-07-14 18:22:47.527880848 +0000 UTC m=+69.065328635" Jul 14 18:22:47.566658 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.566398 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-sd5kk\"" Jul 14 18:22:47.571256 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.571178 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" Jul 14 18:22:47.593531 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.592881 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68877f6bf8-lg2z8"] Jul 14 18:22:47.599322 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:47.599251 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e730311_d011_495f_a4c7_a0097413011c.slice/crio-5bb421857cb1d54794f198e869bb2bca14009f996875f1cdecb61891520e3b9f WatchSource:0}: Error finding container 5bb421857cb1d54794f198e869bb2bca14009f996875f1cdecb61891520e3b9f: Status 404 returned error can't find the container with id 5bb421857cb1d54794f198e869bb2bca14009f996875f1cdecb61891520e3b9f Jul 14 18:22:47.609889 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.609614 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wk6vb\"" Jul 14 18:22:47.614664 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.614370 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:47.655424 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.655314 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59f484b746-8hs6t"] Jul 14 18:22:47.669525 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:47.669498 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cf164f_807a_4191_acdd_e7d5f3c6de75.slice/crio-42c2d2996b8e950012036a489313c4e17cabe6741e3accd7800db413376e726d WatchSource:0}: Error finding container 42c2d2996b8e950012036a489313c4e17cabe6741e3accd7800db413376e726d: Status 404 returned error can't find the container with id 42c2d2996b8e950012036a489313c4e17cabe6741e3accd7800db413376e726d Jul 14 18:22:47.702209 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.702153 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb"] Jul 14 18:22:47.776307 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.773950 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr"] Jul 14 18:22:47.821434 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:47.821404 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7v79r"] Jul 14 18:22:47.826537 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:47.826504 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f139ad7_4fe0_4765_9a55_18231d6dd7cd.slice/crio-0b4407aa64bc7bf910324c224f339ee8a72cf5df77e3b34b4d9cb1ddfb711580 WatchSource:0}: Error finding container 0b4407aa64bc7bf910324c224f339ee8a72cf5df77e3b34b4d9cb1ddfb711580: Status 404 returned error can't find the container with id 0b4407aa64bc7bf910324c224f339ee8a72cf5df77e3b34b4d9cb1ddfb711580 Jul 14 18:22:48.514515 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:48.514423 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" event={"ID":"24acf903-caba-43ed-bd46-cf61a79524c9","Type":"ContainerStarted","Data":"68d6e14942ac3517c4c33cd00e5e8fa928f23678016e9598d8a759663c6c1855"} Jul 14 18:22:48.516767 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:48.516738 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7v79r" event={"ID":"3f139ad7-4fe0-4765-9a55-18231d6dd7cd","Type":"ContainerStarted","Data":"0b4407aa64bc7bf910324c224f339ee8a72cf5df77e3b34b4d9cb1ddfb711580"} Jul 14 18:22:48.519934 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:48.519066 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59f484b746-8hs6t" event={"ID":"c7cf164f-807a-4191-acdd-e7d5f3c6de75","Type":"ContainerStarted","Data":"ef5421f74673d952ca6c464f091ffd44f683243dd372982d8822ca9a09a411a5"} Jul 14 18:22:48.519934 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:48.519110 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59f484b746-8hs6t" event={"ID":"c7cf164f-807a-4191-acdd-e7d5f3c6de75","Type":"ContainerStarted","Data":"42c2d2996b8e950012036a489313c4e17cabe6741e3accd7800db413376e726d"} Jul 14 18:22:48.522660 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:48.522600 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" event={"ID":"4e730311-d011-495f-a4c7-a0097413011c","Type":"ContainerStarted","Data":"ae26cefa60abc3c46c85366750a23230186dbfab6781d82d6742f6f2fd72b042"} Jul 14 18:22:48.522660 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:48.522643 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" event={"ID":"4e730311-d011-495f-a4c7-a0097413011c","Type":"ContainerStarted","Data":"5bb421857cb1d54794f198e869bb2bca14009f996875f1cdecb61891520e3b9f"} Jul 14 18:22:48.523215 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:48.523164 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:22:48.524978 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:48.524921 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" event={"ID":"af335ced-5db5-4190-832e-8ad80e24915b","Type":"ContainerStarted","Data":"9072ad9919b1b2947aea1b4209ef2f3d13111e911187aae020ab0fc76ea29491"} Jul 14 18:22:48.577830 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:48.577768 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-59f484b746-8hs6t" podStartSLOduration=46.577746374 podStartE2EDuration="46.577746374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:48.54867134 +0000 UTC m=+70.086119136" watchObservedRunningTime="2026-07-14 18:22:48.577746374 +0000 UTC m=+70.115194160" Jul 14 18:22:48.578465 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:48.578428 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" podStartSLOduration=69.578416641 podStartE2EDuration="1m9.578416641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:48.577274818 +0000 UTC m=+70.114722603" watchObservedRunningTime="2026-07-14 18:22:48.578416641 +0000 UTC m=+70.115864427" Jul 14 18:22:49.473381 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:49.473303 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:49.477034 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:49.476822 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:49.529820 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:49.529747 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:49.531101 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:49.531072 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-59f484b746-8hs6t" Jul 14 18:22:50.215629 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:50.215604 2573 scope.go:117] "RemoveContainer" containerID="7b1e154b809a1bf0c6d271a5980a6c5a3238de5c628fae884815a134c18c38ca" Jul 14 18:22:53.554212 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:53.554188 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:22:53.554545 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:53.554308 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-mwmlt" event={"ID":"159e6d97-7ed3-42d8-b502-9add9b40eccb","Type":"ContainerStarted","Data":"2b1e129d3e02b52a12bba91ca9b5b9384d3289740e954bad0134671db6939ff5"} Jul 14 18:22:53.781027 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:53.778034 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:53.902817 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:53.902724 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:54.010430 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.010401 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:54.410615 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.410571 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:54.565432 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.565397 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7v79r" event={"ID":"3f139ad7-4fe0-4765-9a55-18231d6dd7cd","Type":"ContainerStarted","Data":"a9ee29d81d20642551162feecc17e1e1c08dd18a77cb9d06bae543e0daa8706a"} Jul 14 18:22:54.565839 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.565462 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7v79r" event={"ID":"3f139ad7-4fe0-4765-9a55-18231d6dd7cd","Type":"ContainerStarted","Data":"7a5a395d07ac66603ae5215745500fa580c5936279288964b515a6318ebaa798"} Jul 14 18:22:54.565839 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.565492 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7v79r" Jul 14 18:22:54.568314 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.568237 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-znj4p" event={"ID":"1091f487-d0e7-4d2d-96ac-1849adce0fa6","Type":"ContainerStarted","Data":"be44112dfeb2eea72bd2821f57a94d2e4bf72060b31d37a00b633b2b8519469a"} Jul 14 18:22:54.570019 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.569989 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" event={"ID":"24acf903-caba-43ed-bd46-cf61a79524c9","Type":"ContainerStarted","Data":"640b38fe7270255201c692449b02dd97811cdd44e33dbd6a768518a2b8afe69c"} Jul 14 18:22:54.571927 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.571898 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gm8z4" event={"ID":"0049cf44-b6fe-4c7c-8e53-adbd70b7eb46","Type":"ContainerStarted","Data":"e930bbd5f01d9f56bee8132e5181858d2b6f89bd8249ae5aa51aa7edf6307268"} Jul 14 18:22:54.573952 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.573928 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29dsq" event={"ID":"2a487018-2ca6-4484-892b-8998a760a384","Type":"ContainerStarted","Data":"98a270a58525a96e1b54577eb41003df4961e228454e508f24ce15c357e8fac7"} Jul 14 18:22:54.574053 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.573978 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29dsq" event={"ID":"2a487018-2ca6-4484-892b-8998a760a384","Type":"ContainerStarted","Data":"0960d7f2459c9602d09ef68542860acefd720be233d6a1dc3f3c0ae220fd218f"} Jul 14 18:22:54.604856 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.604719 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-rxmhj"] Jul 14 18:22:54.645157 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.645123 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-rxmhj" Jul 14 18:22:54.668597 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.668514 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9tfmg\"" Jul 14 18:22:54.668757 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.668613 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 14 18:22:54.669543 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.669474 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-rxmhj"] Jul 14 18:22:54.696488 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.696404 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:54.700479 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.700429 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7v79r" podStartSLOduration=34.051195361 podStartE2EDuration="39.700410388s" totalImagesPullingTime="5.649215027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:54.692004939 +0000 UTC m=+76.229452733" watchObservedRunningTime="2026-07-14 18:22:54.700410388 +0000 UTC m=+76.237858174" Jul 14 18:22:54.745141 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.745109 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/07d20ec4-fefa-4651-ab31-5007b4c91d53-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-rxmhj\" (UID: \"07d20ec4-fefa-4651-ab31-5007b4c91d53\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-rxmhj" Jul 14 18:22:54.797611 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.797548 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-znj4p" podStartSLOduration=17.655508282 podStartE2EDuration="27.797522829s" totalImagesPullingTime="10.142014547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:22:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:54.780593007 +0000 UTC m=+76.318040792" watchObservedRunningTime="2026-07-14 18:22:54.797522829 +0000 UTC m=+76.334970592" Jul 14 18:22:54.846435 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.846399 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/07d20ec4-fefa-4651-ab31-5007b4c91d53-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-rxmhj\" (UID: \"07d20ec4-fefa-4651-ab31-5007b4c91d53\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-rxmhj" Jul 14 18:22:54.849673 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.849627 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/07d20ec4-fefa-4651-ab31-5007b4c91d53-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-rxmhj\" (UID: \"07d20ec4-fefa-4651-ab31-5007b4c91d53\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-rxmhj" Jul 14 18:22:54.860380 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.858700 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-29dsq" podStartSLOduration=69.156640795 podStartE2EDuration="1m15.858681913s" totalImagesPullingTime="6.702041118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:54.857708359 +0000 UTC m=+76.395156144" watchObservedRunningTime="2026-07-14 18:22:54.858681913 +0000 UTC m=+76.396129697" Jul 14 18:22:54.914480 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.914421 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-mwmlt" podStartSLOduration=43.093369072 podStartE2EDuration="52.914399925s" totalImagesPullingTime="9.821030853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:54.913540704 +0000 UTC m=+76.450988488" watchObservedRunningTime="2026-07-14 18:22:54.914399925 +0000 UTC m=+76.451847711" Jul 14 18:22:54.915545 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.915521 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:54.943024 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.942993 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-4t7qb"] Jul 14 18:22:54.957810 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.957778 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-rxmhj" Jul 14 18:22:54.957961 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.957852 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-l4fpb" podStartSLOduration=47.193140148 podStartE2EDuration="52.957832321s" totalImagesPullingTime="5.764692173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:54.956459406 +0000 UTC m=+76.493907193" watchObservedRunningTime="2026-07-14 18:22:54.957832321 +0000 UTC m=+76.495280109" Jul 14 18:22:54.963106 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.963081 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-4t7qb" Jul 14 18:22:54.967130 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.967103 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 18:22:54.967346 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.967262 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 18:22:54.967596 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.967579 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6dh74\"" Jul 14 18:22:54.981169 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:54.981146 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-4t7qb"] Jul 14 18:22:55.039542 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:55.039484 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gm8z4" podStartSLOduration=69.455706317 podStartE2EDuration="1m16.039466232s" totalImagesPullingTime="6.583759915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:55.038272647 +0000 UTC m=+76.575720432" watchObservedRunningTime="2026-07-14 18:22:55.039466232 +0000 UTC m=+76.576913995" Jul 14 18:22:55.048987 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:55.048962 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r4mk\" (UniqueName: \"kubernetes.io/projected/793e27fb-3087-4865-8e9d-f0461c048e1a-kube-api-access-2r4mk\") pod \"downloads-68c9cbb497-4t7qb\" (UID: \"793e27fb-3087-4865-8e9d-f0461c048e1a\") " pod="openshift-console/downloads-68c9cbb497-4t7qb" Jul 14 18:22:55.061715 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:55.061692 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:55.149969 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:55.149933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r4mk\" (UniqueName: \"kubernetes.io/projected/793e27fb-3087-4865-8e9d-f0461c048e1a-kube-api-access-2r4mk\") pod \"downloads-68c9cbb497-4t7qb\" (UID: \"793e27fb-3087-4865-8e9d-f0461c048e1a\") " pod="openshift-console/downloads-68c9cbb497-4t7qb" Jul 14 18:22:55.178333 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:55.178302 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r4mk\" (UniqueName: \"kubernetes.io/projected/793e27fb-3087-4865-8e9d-f0461c048e1a-kube-api-access-2r4mk\") pod \"downloads-68c9cbb497-4t7qb\" (UID: \"793e27fb-3087-4865-8e9d-f0461c048e1a\") " pod="openshift-console/downloads-68c9cbb497-4t7qb" Jul 14 18:22:55.207033 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:55.206994 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-rxmhj"] Jul 14 18:22:55.210746 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:55.210715 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d20ec4_fefa_4651_ab31_5007b4c91d53.slice/crio-e5993385502b42b10aaf7efd8ad315569c44271c01e004121ace9c6e0ecff4d0 WatchSource:0}: Error finding container e5993385502b42b10aaf7efd8ad315569c44271c01e004121ace9c6e0ecff4d0: Status 404 returned error can't find the container with id e5993385502b42b10aaf7efd8ad315569c44271c01e004121ace9c6e0ecff4d0 Jul 14 18:22:55.263956 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:55.263926 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:55.274992 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:55.274967 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-4t7qb" Jul 14 18:22:55.373588 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:55.373548 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:55.575663 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:55.575634 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-4t7qb"] Jul 14 18:22:55.579370 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:22:55.579339 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793e27fb_3087_4865_8e9d_f0461c048e1a.slice/crio-2186ccb7ca7245d72cb4c671d2bd96aa42c31359dcd6e4e51b4d08df74d9bfbd WatchSource:0}: Error finding container 2186ccb7ca7245d72cb4c671d2bd96aa42c31359dcd6e4e51b4d08df74d9bfbd: Status 404 returned error can't find the container with id 2186ccb7ca7245d72cb4c671d2bd96aa42c31359dcd6e4e51b4d08df74d9bfbd Jul 14 18:22:55.581296 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:55.581244 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" event={"ID":"af335ced-5db5-4190-832e-8ad80e24915b","Type":"ContainerStarted","Data":"63ec5417ee47bf1fccd20d87154b5b0a0e060228bf8bef69e5fe2841c47d3c30"} Jul 14 18:22:55.582621 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:55.582595 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-rxmhj" event={"ID":"07d20ec4-fefa-4651-ab31-5007b4c91d53","Type":"ContainerStarted","Data":"e5993385502b42b10aaf7efd8ad315569c44271c01e004121ace9c6e0ecff4d0"} Jul 14 18:22:55.611581 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:55.611549 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:55.846927 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:55.846899 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:55.967671 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:55.967597 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:56.066244 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:56.066213 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:56.215191 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:56.215153 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:56.437513 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:56.437434 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:56.588466 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:56.588415 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-4t7qb" event={"ID":"793e27fb-3087-4865-8e9d-f0461c048e1a","Type":"ContainerStarted","Data":"2186ccb7ca7245d72cb4c671d2bd96aa42c31359dcd6e4e51b4d08df74d9bfbd"} Jul 14 18:22:56.590497 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:56.590465 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" event={"ID":"af335ced-5db5-4190-832e-8ad80e24915b","Type":"ContainerStarted","Data":"ea4556ff8a8d5d8b5e5d9c976998205774eb2b8fe0d86d10f544bb5b14ddf8ea"} Jul 14 18:22:56.613137 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:56.613087 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-bhjwr" podStartSLOduration=47.026968848 podStartE2EDuration="54.613071418s" totalImagesPullingTime="7.58610257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:56.612434424 +0000 UTC m=+78.149882220" watchObservedRunningTime="2026-07-14 18:22:56.613071418 +0000 UTC m=+78.150519202" Jul 14 18:22:56.851595 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:56.851551 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:56.994719 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:56.994686 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:58.042554 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:58.042487 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:22:59.605927 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:59.605839 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-rxmhj" event={"ID":"07d20ec4-fefa-4651-ab31-5007b4c91d53","Type":"ContainerStarted","Data":"2a568187d270e622d13525d023244947e967258d8360c38b9a8b970fb6bc4689"} Jul 14 18:22:59.606357 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:59.606205 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-rxmhj" Jul 14 18:22:59.611689 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:59.611663 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-rxmhj" Jul 14 18:22:59.625212 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:22:59.625147 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-rxmhj" podStartSLOduration=1.565921915 podStartE2EDuration="5.625128653s" totalImagesPullingTime="4.059206738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:59.624661177 +0000 UTC m=+81.162108963" watchObservedRunningTime="2026-07-14 18:22:59.625128653 +0000 UTC m=+81.162576438" Jul 14 18:23:00.632303 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:00.632143 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:00.703460 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:00.703418 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:00.968834 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:00.968792 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:03.409278 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:03.409242 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:03.522433 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:03.522373 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:03.619742 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:03.619707 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:03.921506 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:03.921464 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:04.574568 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:04.574534 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:23:04.578650 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:04.578628 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-mwmlt" Jul 14 18:23:04.585702 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:04.585680 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7v79r" Jul 14 18:23:04.839010 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:04.838884 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:05.693956 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:05.693910 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:05.849813 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:05.849765 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:05.974419 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:05.974344 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd"] Jul 14 18:23:05.978132 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:05.978090 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" Jul 14 18:23:05.979417 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:05.979343 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:05.983205 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:05.983173 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-82z45\"" Jul 14 18:23:05.983337 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:05.983236 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 18:23:05.985446 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:05.985258 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 14 18:23:05.985446 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:05.985315 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 14 18:23:06.002746 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.002669 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd"] Jul 14 18:23:06.013969 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.013940 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-hbr4k"] Jul 14 18:23:06.016449 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.016429 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nsfmd"] Jul 14 18:23:06.016571 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.016551 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.019229 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.019192 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.020167 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.020145 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 14 18:23:06.020262 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.020198 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xgh42\"" Jul 14 18:23:06.021806 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.021784 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 14 18:23:06.022574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.022544 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h2g88\"" Jul 14 18:23:06.023159 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.023021 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 18:23:06.023159 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.023028 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 18:23:06.023159 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.023031 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 18:23:06.029697 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.029676 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 14 18:23:06.041793 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.041763 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07fada97-ddc6-4d47-b0c3-5d7816ea4027-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-q9ppd\" (UID: \"07fada97-ddc6-4d47-b0c3-5d7816ea4027\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" Jul 14 18:23:06.041898 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.041809 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07fada97-ddc6-4d47-b0c3-5d7816ea4027-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-q9ppd\" (UID: \"07fada97-ddc6-4d47-b0c3-5d7816ea4027\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" Jul 14 18:23:06.041898 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.041827 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/264c6200-8285-41b2-a645-5a025b3a7c6f-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.041898 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.041891 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/264c6200-8285-41b2-a645-5a025b3a7c6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.042039 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.041921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/264c6200-8285-41b2-a645-5a025b3a7c6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.042039 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.041962 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msm2l\" (UniqueName: \"kubernetes.io/projected/264c6200-8285-41b2-a645-5a025b3a7c6f-kube-api-access-msm2l\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.042039 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.042020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/264c6200-8285-41b2-a645-5a025b3a7c6f-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.042144 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.042058 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/264c6200-8285-41b2-a645-5a025b3a7c6f-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.042144 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.042108 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48b76\" (UniqueName: \"kubernetes.io/projected/07fada97-ddc6-4d47-b0c3-5d7816ea4027-kube-api-access-48b76\") pod \"openshift-state-metrics-84bd68b896-q9ppd\" (UID: \"07fada97-ddc6-4d47-b0c3-5d7816ea4027\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" Jul 14 18:23:06.042214 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.042140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/07fada97-ddc6-4d47-b0c3-5d7816ea4027-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-q9ppd\" (UID: \"07fada97-ddc6-4d47-b0c3-5d7816ea4027\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" Jul 14 18:23:06.048904 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.048877 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-hbr4k"] Jul 14 18:23:06.106450 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.106402 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:06.142728 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.142693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64ba4ec1-e886-4191-b8ad-15606471581b-metrics-client-ca\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.142894 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.142752 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48b76\" (UniqueName: \"kubernetes.io/projected/07fada97-ddc6-4d47-b0c3-5d7816ea4027-kube-api-access-48b76\") pod \"openshift-state-metrics-84bd68b896-q9ppd\" (UID: \"07fada97-ddc6-4d47-b0c3-5d7816ea4027\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" Jul 14 18:23:06.142894 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.142787 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/07fada97-ddc6-4d47-b0c3-5d7816ea4027-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-q9ppd\" (UID: \"07fada97-ddc6-4d47-b0c3-5d7816ea4027\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" Jul 14 18:23:06.142894 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.142818 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/64ba4ec1-e886-4191-b8ad-15606471581b-node-exporter-accelerators-collector-config\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.142894 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.142850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64ba4ec1-e886-4191-b8ad-15606471581b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.142894 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.142879 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07fada97-ddc6-4d47-b0c3-5d7816ea4027-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-q9ppd\" (UID: \"07fada97-ddc6-4d47-b0c3-5d7816ea4027\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" Jul 14 18:23:06.143168 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.142908 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07fada97-ddc6-4d47-b0c3-5d7816ea4027-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-q9ppd\" (UID: \"07fada97-ddc6-4d47-b0c3-5d7816ea4027\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" Jul 14 18:23:06.143168 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:23:06.142937 2573 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 14 18:23:06.143168 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.142958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/264c6200-8285-41b2-a645-5a025b3a7c6f-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.143168 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.142998 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/64ba4ec1-e886-4191-b8ad-15606471581b-node-exporter-wtmp\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.143168 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.143035 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64ba4ec1-e886-4191-b8ad-15606471581b-sys\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.143168 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:23:06.143038 2573 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 14 18:23:06.143168 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:23:06.143058 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07fada97-ddc6-4d47-b0c3-5d7816ea4027-openshift-state-metrics-tls podName:07fada97-ddc6-4d47-b0c3-5d7816ea4027 nodeName:}" failed. No retries permitted until 2026-07-14 18:23:06.643038955 +0000 UTC m=+88.180486721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/07fada97-ddc6-4d47-b0c3-5d7816ea4027-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-q9ppd" (UID: "07fada97-ddc6-4d47-b0c3-5d7816ea4027") : secret "openshift-state-metrics-tls" not found Jul 14 18:23:06.143168 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:23:06.143095 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264c6200-8285-41b2-a645-5a025b3a7c6f-kube-state-metrics-tls podName:264c6200-8285-41b2-a645-5a025b3a7c6f nodeName:}" failed. No retries permitted until 2026-07-14 18:23:06.64307731 +0000 UTC m=+88.180525094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/264c6200-8285-41b2-a645-5a025b3a7c6f-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-hbr4k" (UID: "264c6200-8285-41b2-a645-5a025b3a7c6f") : secret "kube-state-metrics-tls" not found Jul 14 18:23:06.143662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.143171 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/264c6200-8285-41b2-a645-5a025b3a7c6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.143662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.143212 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sktkm\" (UniqueName: \"kubernetes.io/projected/64ba4ec1-e886-4191-b8ad-15606471581b-kube-api-access-sktkm\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.143662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.143244 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/264c6200-8285-41b2-a645-5a025b3a7c6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.143662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.143312 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/64ba4ec1-e886-4191-b8ad-15606471581b-root\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.143662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.143357 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msm2l\" (UniqueName: \"kubernetes.io/projected/264c6200-8285-41b2-a645-5a025b3a7c6f-kube-api-access-msm2l\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.143662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.143387 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/264c6200-8285-41b2-a645-5a025b3a7c6f-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.143662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.143494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/64ba4ec1-e886-4191-b8ad-15606471581b-node-exporter-tls\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.143662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.143528 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/64ba4ec1-e886-4191-b8ad-15606471581b-node-exporter-textfile\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.143662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.143568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/264c6200-8285-41b2-a645-5a025b3a7c6f-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.144109 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.143897 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07fada97-ddc6-4d47-b0c3-5d7816ea4027-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-q9ppd\" (UID: \"07fada97-ddc6-4d47-b0c3-5d7816ea4027\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" Jul 14 18:23:06.144109 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.143952 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/264c6200-8285-41b2-a645-5a025b3a7c6f-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.144109 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.144091 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/264c6200-8285-41b2-a645-5a025b3a7c6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.144253 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.144134 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/264c6200-8285-41b2-a645-5a025b3a7c6f-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.145942 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.145887 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/264c6200-8285-41b2-a645-5a025b3a7c6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.145942 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.145888 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07fada97-ddc6-4d47-b0c3-5d7816ea4027-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-q9ppd\" (UID: \"07fada97-ddc6-4d47-b0c3-5d7816ea4027\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" Jul 14 18:23:06.154782 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.154761 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48b76\" (UniqueName: \"kubernetes.io/projected/07fada97-ddc6-4d47-b0c3-5d7816ea4027-kube-api-access-48b76\") pod \"openshift-state-metrics-84bd68b896-q9ppd\" (UID: \"07fada97-ddc6-4d47-b0c3-5d7816ea4027\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" Jul 14 18:23:06.156771 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.156746 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msm2l\" (UniqueName: \"kubernetes.io/projected/264c6200-8285-41b2-a645-5a025b3a7c6f-kube-api-access-msm2l\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.244349 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.244254 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/64ba4ec1-e886-4191-b8ad-15606471581b-node-exporter-accelerators-collector-config\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.244349 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.244320 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64ba4ec1-e886-4191-b8ad-15606471581b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.244571 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.244370 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/64ba4ec1-e886-4191-b8ad-15606471581b-node-exporter-wtmp\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.244571 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.244403 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64ba4ec1-e886-4191-b8ad-15606471581b-sys\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.244571 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.244437 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktkm\" (UniqueName: \"kubernetes.io/projected/64ba4ec1-e886-4191-b8ad-15606471581b-kube-api-access-sktkm\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.244571 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.244478 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/64ba4ec1-e886-4191-b8ad-15606471581b-root\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.244571 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.244518 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/64ba4ec1-e886-4191-b8ad-15606471581b-node-exporter-tls\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.244571 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.244530 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64ba4ec1-e886-4191-b8ad-15606471581b-sys\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.244571 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.244547 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/64ba4ec1-e886-4191-b8ad-15606471581b-node-exporter-textfile\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.244571 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.244549 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/64ba4ec1-e886-4191-b8ad-15606471581b-node-exporter-wtmp\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.244994 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.244609 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64ba4ec1-e886-4191-b8ad-15606471581b-metrics-client-ca\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.244994 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.244789 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/64ba4ec1-e886-4191-b8ad-15606471581b-root\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.244994 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.244817 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/64ba4ec1-e886-4191-b8ad-15606471581b-node-exporter-textfile\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.244994 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.244941 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/64ba4ec1-e886-4191-b8ad-15606471581b-node-exporter-accelerators-collector-config\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.245182 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.245143 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64ba4ec1-e886-4191-b8ad-15606471581b-metrics-client-ca\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.247027 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.247004 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64ba4ec1-e886-4191-b8ad-15606471581b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.247400 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.247376 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/64ba4ec1-e886-4191-b8ad-15606471581b-node-exporter-tls\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.257743 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.257718 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sktkm\" (UniqueName: \"kubernetes.io/projected/64ba4ec1-e886-4191-b8ad-15606471581b-kube-api-access-sktkm\") pod \"node-exporter-nsfmd\" (UID: \"64ba4ec1-e886-4191-b8ad-15606471581b\") " pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.306942 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.306903 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:06.336127 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.336091 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nsfmd" Jul 14 18:23:06.406362 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.406306 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:06.507531 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.507449 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:06.649255 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.649211 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/264c6200-8285-41b2-a645-5a025b3a7c6f-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.649476 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.649455 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/07fada97-ddc6-4d47-b0c3-5d7816ea4027-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-q9ppd\" (UID: \"07fada97-ddc6-4d47-b0c3-5d7816ea4027\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" Jul 14 18:23:06.652191 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.652169 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/264c6200-8285-41b2-a645-5a025b3a7c6f-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-hbr4k\" (UID: \"264c6200-8285-41b2-a645-5a025b3a7c6f\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:06.652337 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.652222 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/07fada97-ddc6-4d47-b0c3-5d7816ea4027-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-q9ppd\" (UID: \"07fada97-ddc6-4d47-b0c3-5d7816ea4027\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" Jul 14 18:23:06.890326 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.890215 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" Jul 14 18:23:06.908248 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.908212 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:06.928881 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:06.928849 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" Jul 14 18:23:07.067739 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.067705 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:07.112025 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.111992 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 18:23:07.116208 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.116184 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.119233 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.119197 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 14 18:23:07.119880 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.119860 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 14 18:23:07.119972 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.119905 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 14 18:23:07.121210 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.121189 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 14 18:23:07.121898 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.121862 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 14 18:23:07.122006 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.121989 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c5l48\"" Jul 14 18:23:07.122075 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.122008 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 14 18:23:07.122896 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.122880 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 14 18:23:07.122996 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.122902 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 14 18:23:07.126566 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.126543 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 14 18:23:07.144557 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.144483 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 18:23:07.154385 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.154353 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.154524 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.154397 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.154524 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.154422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.154524 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.154491 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pjnp\" (UniqueName: \"kubernetes.io/projected/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-kube-api-access-5pjnp\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.154524 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.154518 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-web-config\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.154734 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.154550 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.154734 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.154619 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-config-volume\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.154734 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.154680 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.154734 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.154707 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.154926 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.154738 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.154926 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.154796 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.154926 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.154821 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-config-out\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.154926 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.154842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.156664 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.156637 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:07.255413 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.255378 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-config-volume\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.255607 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.255439 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.255607 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.255469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.255607 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.255495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.255607 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.255534 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.255607 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.255563 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-config-out\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.255607 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.255598 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.255917 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.255632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.255917 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.255648 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.255917 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.255663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.255917 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.255690 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjnp\" (UniqueName: \"kubernetes.io/projected/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-kube-api-access-5pjnp\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.255917 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.255708 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-web-config\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.255917 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.255728 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.256152 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:23:07.255918 2573 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 14 18:23:07.256152 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:23:07.255996 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-main-tls podName:fe46bfc2-36dd-491f-9b6b-5b7bbea27086 nodeName:}" failed. No retries permitted until 2026-07-14 18:23:07.75597409 +0000 UTC m=+89.293421867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "fe46bfc2-36dd-491f-9b6b-5b7bbea27086") : secret "alertmanager-main-tls" not found Jul 14 18:23:07.256874 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.256575 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.256874 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.256749 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.258796 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.258704 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.258796 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.258706 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-config-volume\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.259275 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.259227 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-config-out\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.259395 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.259342 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.259579 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.259534 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.259818 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.259774 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.260380 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.260336 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.260824 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.260796 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.262047 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.262029 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-web-config\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.266091 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.266069 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjnp\" (UniqueName: \"kubernetes.io/projected/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-kube-api-access-5pjnp\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.468611 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.468565 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:07.760534 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.760446 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:07.763081 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:07.763057 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:08.029798 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.029712 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:23:08.042100 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.042002 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-797996db74-bzk2c"] Jul 14 18:23:08.046344 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.046320 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.050090 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.050064 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 14 18:23:08.050346 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.050326 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 14 18:23:08.050517 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.050449 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-l2zkr\"" Jul 14 18:23:08.050636 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.050529 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 14 18:23:08.050980 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.050938 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 14 18:23:08.050980 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.050952 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eta1e2g1oioau\"" Jul 14 18:23:08.051325 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.051304 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 14 18:23:08.062467 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.062395 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-797996db74-bzk2c"] Jul 14 18:23:08.096521 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.096443 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:08.164083 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.164044 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.164230 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.164098 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.164230 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.164164 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.164230 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.164185 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cbd1903-2698-4c5d-8e73-2910b6991f36-metrics-client-ca\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.164230 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.164212 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8rhm\" (UniqueName: \"kubernetes.io/projected/8cbd1903-2698-4c5d-8e73-2910b6991f36-kube-api-access-x8rhm\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.164465 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.164329 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.164465 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.164392 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-thanos-querier-tls\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.164465 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.164421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-grpc-tls\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.201931 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.201872 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:08.265879 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.265844 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.266054 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.265897 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-thanos-querier-tls\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.266054 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.266031 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-grpc-tls\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.266163 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.266148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.266222 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.266187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.266275 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.266218 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.266275 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.266245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cbd1903-2698-4c5d-8e73-2910b6991f36-metrics-client-ca\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.266410 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.266271 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8rhm\" (UniqueName: \"kubernetes.io/projected/8cbd1903-2698-4c5d-8e73-2910b6991f36-kube-api-access-x8rhm\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.267277 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.267223 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cbd1903-2698-4c5d-8e73-2910b6991f36-metrics-client-ca\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.269708 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.269662 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.269708 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.269677 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-thanos-querier-tls\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.269861 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.269712 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.270058 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.270013 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.270058 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.270049 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.270649 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.270624 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8cbd1903-2698-4c5d-8e73-2910b6991f36-secret-grpc-tls\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.277513 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.277492 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8rhm\" (UniqueName: \"kubernetes.io/projected/8cbd1903-2698-4c5d-8e73-2910b6991f36-kube-api-access-x8rhm\") pod \"thanos-querier-797996db74-bzk2c\" (UID: \"8cbd1903-2698-4c5d-8e73-2910b6991f36\") " pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.360554 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.360468 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:08.708634 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:08.708596 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:10.539339 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:10.539305 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:23:10.716389 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:10.716340 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd"] Jul 14 18:23:10.720192 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:10.720114 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd" Jul 14 18:23:10.723595 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:10.723572 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 14 18:23:10.723739 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:10.723630 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mg2x6\"" Jul 14 18:23:10.728765 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:10.728738 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd"] Jul 14 18:23:10.795501 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:10.795414 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9116c6f0-66e8-4503-8570-fd20f75816cb-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-z9kjd\" (UID: \"9116c6f0-66e8-4503-8570-fd20f75816cb\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd" Jul 14 18:23:10.795676 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:10.795532 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bddxp\" (UniqueName: \"kubernetes.io/projected/9116c6f0-66e8-4503-8570-fd20f75816cb-kube-api-access-bddxp\") pod \"monitoring-plugin-5f68df7f58-z9kjd\" (UID: \"9116c6f0-66e8-4503-8570-fd20f75816cb\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd" Jul 14 18:23:10.896964 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:10.896929 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bddxp\" (UniqueName: \"kubernetes.io/projected/9116c6f0-66e8-4503-8570-fd20f75816cb-kube-api-access-bddxp\") pod \"monitoring-plugin-5f68df7f58-z9kjd\" (UID: \"9116c6f0-66e8-4503-8570-fd20f75816cb\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd" Jul 14 18:23:10.897153 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:10.896987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9116c6f0-66e8-4503-8570-fd20f75816cb-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-z9kjd\" (UID: \"9116c6f0-66e8-4503-8570-fd20f75816cb\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd" Jul 14 18:23:10.897153 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:23:10.897135 2573 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 14 18:23:10.897260 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:23:10.897204 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9116c6f0-66e8-4503-8570-fd20f75816cb-monitoring-plugin-cert podName:9116c6f0-66e8-4503-8570-fd20f75816cb nodeName:}" failed. No retries permitted until 2026-07-14 18:23:11.397185773 +0000 UTC m=+92.934633537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9116c6f0-66e8-4503-8570-fd20f75816cb-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-z9kjd" (UID: "9116c6f0-66e8-4503-8570-fd20f75816cb") : secret "monitoring-plugin-cert" not found Jul 14 18:23:10.912042 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:10.912007 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bddxp\" (UniqueName: \"kubernetes.io/projected/9116c6f0-66e8-4503-8570-fd20f75816cb-kube-api-access-bddxp\") pod \"monitoring-plugin-5f68df7f58-z9kjd\" (UID: \"9116c6f0-66e8-4503-8570-fd20f75816cb\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd" Jul 14 18:23:11.321918 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:11.321880 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:11.403470 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:11.403427 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9116c6f0-66e8-4503-8570-fd20f75816cb-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-z9kjd\" (UID: \"9116c6f0-66e8-4503-8570-fd20f75816cb\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd" Jul 14 18:23:11.406220 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:11.406190 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9116c6f0-66e8-4503-8570-fd20f75816cb-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-z9kjd\" (UID: \"9116c6f0-66e8-4503-8570-fd20f75816cb\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd" Jul 14 18:23:11.420730 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:11.420685 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:11.633427 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:11.633334 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd" Jul 14 18:23:12.161153 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:12.161115 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:12.452247 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:12.452215 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:12.573167 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:12.573130 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:13.486213 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:13.486183 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd"] Jul 14 18:23:13.517435 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:13.517405 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 18:23:13.639932 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:23:13.639861 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fada97_ddc6_4d47_b0c3_5d7816ea4027.slice/crio-d880768f1062ce1f458e17ef67cc1b06fc4d55d0b73e3b7e236d7a60e6dc6db3 WatchSource:0}: Error finding container d880768f1062ce1f458e17ef67cc1b06fc4d55d0b73e3b7e236d7a60e6dc6db3: Status 404 returned error can't find the container with id d880768f1062ce1f458e17ef67cc1b06fc4d55d0b73e3b7e236d7a60e6dc6db3 Jul 14 18:23:13.640805 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:23:13.640688 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe46bfc2_36dd_491f_9b6b_5b7bbea27086.slice/crio-6866e570e5eb20b0a0ac323c1061a43dcdcb8041bb74ecc7c8d33b5b7ee6d08a WatchSource:0}: Error finding container 6866e570e5eb20b0a0ac323c1061a43dcdcb8041bb74ecc7c8d33b5b7ee6d08a: Status 404 returned error can't find the container with id 6866e570e5eb20b0a0ac323c1061a43dcdcb8041bb74ecc7c8d33b5b7ee6d08a Jul 14 18:23:13.663909 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:13.663881 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nsfmd" event={"ID":"64ba4ec1-e886-4191-b8ad-15606471581b","Type":"ContainerStarted","Data":"c5a86efa82408437f68149d8ac2ef6cb201d8c93183147695881cab9ce6af60b"} Jul 14 18:23:13.665022 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:13.664988 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe46bfc2-36dd-491f-9b6b-5b7bbea27086","Type":"ContainerStarted","Data":"6866e570e5eb20b0a0ac323c1061a43dcdcb8041bb74ecc7c8d33b5b7ee6d08a"} Jul 14 18:23:13.665988 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:13.665967 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" event={"ID":"07fada97-ddc6-4d47-b0c3-5d7816ea4027","Type":"ContainerStarted","Data":"d880768f1062ce1f458e17ef67cc1b06fc4d55d0b73e3b7e236d7a60e6dc6db3"} Jul 14 18:23:13.735849 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:13.735752 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd"] Jul 14 18:23:13.739073 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:13.738873 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-hbr4k"] Jul 14 18:23:13.741686 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:13.741664 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-797996db74-bzk2c"] Jul 14 18:23:13.745456 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:23:13.745429 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9116c6f0_66e8_4503_8570_fd20f75816cb.slice/crio-7ed82b341b209b11346ddc64038693f23e0866a24309d5415418ec36db6c2e0f WatchSource:0}: Error finding container 7ed82b341b209b11346ddc64038693f23e0866a24309d5415418ec36db6c2e0f: Status 404 returned error can't find the container with id 7ed82b341b209b11346ddc64038693f23e0866a24309d5415418ec36db6c2e0f Jul 14 18:23:13.745771 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:23:13.745739 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbd1903_2698_4c5d_8e73_2910b6991f36.slice/crio-5e6d8761dc113b3ca2bae114fee4adf92227c454ff53d1ef40a6a062211b9306 WatchSource:0}: Error finding container 5e6d8761dc113b3ca2bae114fee4adf92227c454ff53d1ef40a6a062211b9306: Status 404 returned error can't find the container with id 5e6d8761dc113b3ca2bae114fee4adf92227c454ff53d1ef40a6a062211b9306 Jul 14 18:23:13.746111 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:23:13.746094 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264c6200_8285_41b2_a645_5a025b3a7c6f.slice/crio-9d4b92803fb82257ba050803fd46e36b3fdda01a3fc14158984b0eb6f84e6a87 WatchSource:0}: Error finding container 9d4b92803fb82257ba050803fd46e36b3fdda01a3fc14158984b0eb6f84e6a87: Status 404 returned error can't find the container with id 9d4b92803fb82257ba050803fd46e36b3fdda01a3fc14158984b0eb6f84e6a87 Jul 14 18:23:14.680973 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:14.680852 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-4t7qb" event={"ID":"793e27fb-3087-4865-8e9d-f0461c048e1a","Type":"ContainerStarted","Data":"e5c04471697f0f0a18097278098e49833b8f9d631789cf8f5d0f150494214f15"} Jul 14 18:23:14.681569 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:14.681124 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-4t7qb" Jul 14 18:23:14.683930 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:14.683889 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" event={"ID":"8cbd1903-2698-4c5d-8e73-2910b6991f36","Type":"ContainerStarted","Data":"5e6d8761dc113b3ca2bae114fee4adf92227c454ff53d1ef40a6a062211b9306"} Jul 14 18:23:14.686516 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:14.686450 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" event={"ID":"07fada97-ddc6-4d47-b0c3-5d7816ea4027","Type":"ContainerStarted","Data":"37e03cc8c961edc73cfff94e00209430b8adfcfcd74ae7f60597642f3e9ef4c3"} Jul 14 18:23:14.686656 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:14.686516 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" event={"ID":"07fada97-ddc6-4d47-b0c3-5d7816ea4027","Type":"ContainerStarted","Data":"9a30f5bac71d571ae147cb9c73b6f7b495abf1a08360b884a701f750864e9896"} Jul 14 18:23:14.688155 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:14.688102 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd" event={"ID":"9116c6f0-66e8-4503-8570-fd20f75816cb","Type":"ContainerStarted","Data":"7ed82b341b209b11346ddc64038693f23e0866a24309d5415418ec36db6c2e0f"} Jul 14 18:23:14.690187 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:14.690120 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-4t7qb" Jul 14 18:23:14.691170 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:14.691098 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" event={"ID":"264c6200-8285-41b2-a645-5a025b3a7c6f","Type":"ContainerStarted","Data":"9d4b92803fb82257ba050803fd46e36b3fdda01a3fc14158984b0eb6f84e6a87"} Jul 14 18:23:14.712711 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:14.712428 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-4t7qb" podStartSLOduration=2.567447823 podStartE2EDuration="20.712409413s" totalImagesPullingTime="18.14496159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:14.70921743 +0000 UTC m=+96.246665215" watchObservedRunningTime="2026-07-14 18:23:14.712409413 +0000 UTC m=+96.249857199" Jul 14 18:23:15.397343 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:15.397252 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:15.501006 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:15.500968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:15.568528 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:15.568445 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:16.860964 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:16.860892 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:16.994757 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:16.994711 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:18.527933 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:18.527898 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rdlqb" Jul 14 18:23:18.716331 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:18.716181 2573 generic.go:356] "Generic (PLEG): container finished" podID="64ba4ec1-e886-4191-b8ad-15606471581b" containerID="07a7bd0158ee28042c3740d90da5cf515cb74fe71aa530ce5281d793b35f8ba5" exitCode=0 Jul 14 18:23:18.716646 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:18.716572 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nsfmd" event={"ID":"64ba4ec1-e886-4191-b8ad-15606471581b","Type":"ContainerDied","Data":"07a7bd0158ee28042c3740d90da5cf515cb74fe71aa530ce5281d793b35f8ba5"} Jul 14 18:23:18.719574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:18.719544 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd" event={"ID":"9116c6f0-66e8-4503-8570-fd20f75816cb","Type":"ContainerStarted","Data":"4141aa9ca50105747ecfaaa7e9349324cae7e6cdca0e8bba2e4ee5f0de55b111"} Jul 14 18:23:18.719855 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:18.719751 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd" Jul 14 18:23:18.723856 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:18.723827 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" event={"ID":"8cbd1903-2698-4c5d-8e73-2910b6991f36","Type":"ContainerStarted","Data":"4eca97c4e2ff78da126ffa749ddf84afc718fda5483e5a7cf0048a8e9f417ee4"} Jul 14 18:23:18.723957 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:18.723894 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" event={"ID":"8cbd1903-2698-4c5d-8e73-2910b6991f36","Type":"ContainerStarted","Data":"eb55999a0383256b0ce68d0a1e5ba9eaa77fa6200aafcc0321fb0aeefc272524"} Jul 14 18:23:18.723957 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:18.723938 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" event={"ID":"8cbd1903-2698-4c5d-8e73-2910b6991f36","Type":"ContainerStarted","Data":"8020bca2efc10c4b3ec0c07e4b28da0f9ce43f7ba36937fb7af10a2b9244ff97"} Jul 14 18:23:18.726084 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:18.726047 2573 generic.go:356] "Generic (PLEG): container finished" podID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerID="8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e" exitCode=0 Jul 14 18:23:18.726183 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:18.726126 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe46bfc2-36dd-491f-9b6b-5b7bbea27086","Type":"ContainerDied","Data":"8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e"} Jul 14 18:23:18.728769 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:18.728751 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd" Jul 14 18:23:18.735085 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:18.735058 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" event={"ID":"07fada97-ddc6-4d47-b0c3-5d7816ea4027","Type":"ContainerStarted","Data":"4728134669da967a210ee5b6a8b34f2fa620374449606428b5ada18c183e2fbc"} Jul 14 18:23:18.818132 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:18.818079 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-z9kjd" podStartSLOduration=6.090015947 podStartE2EDuration="8.818061472s" totalImagesPullingTime="2.728045525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:18.816643276 +0000 UTC m=+100.354091070" watchObservedRunningTime="2026-07-14 18:23:18.818061472 +0000 UTC m=+100.355509259" Jul 14 18:23:18.861312 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:18.861237 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-q9ppd" podStartSLOduration=9.839710422 podStartE2EDuration="13.861212074s" totalImagesPullingTime="4.021501652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:18.859605239 +0000 UTC m=+100.397053025" watchObservedRunningTime="2026-07-14 18:23:18.861212074 +0000 UTC m=+100.398659860" Jul 14 18:23:19.239811 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:19.239785 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:19.334753 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:19.334670 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:19.752438 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:19.752345 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nsfmd" event={"ID":"64ba4ec1-e886-4191-b8ad-15606471581b","Type":"ContainerStarted","Data":"d70f7b6a88082dcc8f6bd4dcb1d884c0796e0366d4cbef737835f6a32bb38f27"} Jul 14 18:23:19.752937 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:19.752452 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nsfmd" event={"ID":"64ba4ec1-e886-4191-b8ad-15606471581b","Type":"ContainerStarted","Data":"d7d181be3d29c6a8c906f1758e6cfcc08b1f2b5a1a8070e38ac5827b7a76561c"} Jul 14 18:23:19.756271 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:19.756200 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" event={"ID":"264c6200-8285-41b2-a645-5a025b3a7c6f","Type":"ContainerStarted","Data":"a83a28f70ffedd4a09a3aa3c6b02bde63cc2e56b6d5d98d4413ddbf642f0b855"} Jul 14 18:23:19.798013 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:19.797936 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nsfmd" podStartSLOduration=11.515828133 podStartE2EDuration="14.797915034s" totalImagesPullingTime="3.156252976s" totalInitContainerRuntime="125.833925ms" isStatefulPod=true podCreationTimestamp="2026-07-14 18:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:19.796887188 +0000 UTC m=+101.334334977" watchObservedRunningTime="2026-07-14 18:23:19.797915034 +0000 UTC m=+101.335362823" Jul 14 18:23:19.892792 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:19.892473 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68877f6bf8-lg2z8"] Jul 14 18:23:20.009537 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:20.009469 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:20.160179 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:20.160040 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:20.273931 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:20.273896 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:20.766452 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:20.766342 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" event={"ID":"264c6200-8285-41b2-a645-5a025b3a7c6f","Type":"ContainerStarted","Data":"d0e6f1132c25dce872db66aa8251c429efac19b5a539e34064c8294c5634bafe"} Jul 14 18:23:20.766452 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:20.766408 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" event={"ID":"264c6200-8285-41b2-a645-5a025b3a7c6f","Type":"ContainerStarted","Data":"46dbda502a04b26d084bb18eb75f4f2785c5a38109de0a35c734f193e90c3ae7"} Jul 14 18:23:20.799577 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:20.798173 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-hbr4k" podStartSLOduration=10.075943808 podStartE2EDuration="15.798148889s" totalImagesPullingTime="5.722205081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:20.795688669 +0000 UTC m=+102.333136454" watchObservedRunningTime="2026-07-14 18:23:20.798148889 +0000 UTC m=+102.335596675" Jul 14 18:23:21.569675 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:21.569643 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:21.743035 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:21.743006 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:21.773575 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:21.773545 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" event={"ID":"8cbd1903-2698-4c5d-8e73-2910b6991f36","Type":"ContainerStarted","Data":"74b0790cc55fd97641de15c19d0ff9556e827cb73b08baa1c4828305954f7c08"} Jul 14 18:23:21.773956 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:21.773626 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" event={"ID":"8cbd1903-2698-4c5d-8e73-2910b6991f36","Type":"ContainerStarted","Data":"af1d58dfece5be2dde1bed2a5b319ee8a5c4a01de7d63f0b4b1e4261177f2a3f"} Jul 14 18:23:21.773956 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:21.773669 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" event={"ID":"8cbd1903-2698-4c5d-8e73-2910b6991f36","Type":"ContainerStarted","Data":"a22eae2fa31bd30e2127344754c4b3b5c98b5cdaee48f5f2331b5278a298f2ae"} Jul 14 18:23:21.773956 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:21.773716 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:21.782353 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:21.782322 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe46bfc2-36dd-491f-9b6b-5b7bbea27086","Type":"ContainerStarted","Data":"d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850"} Jul 14 18:23:21.782484 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:21.782432 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe46bfc2-36dd-491f-9b6b-5b7bbea27086","Type":"ContainerStarted","Data":"4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292"} Jul 14 18:23:21.847161 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:21.847103 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" podStartSLOduration=7.078106959 podStartE2EDuration="13.847079689s" totalImagesPullingTime="6.76897273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:23:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:21.841705124 +0000 UTC m=+103.379152908" watchObservedRunningTime="2026-07-14 18:23:21.847079689 +0000 UTC m=+103.384527476" Jul 14 18:23:22.792032 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:22.791988 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe46bfc2-36dd-491f-9b6b-5b7bbea27086","Type":"ContainerStarted","Data":"4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61"} Jul 14 18:23:22.792502 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:22.792069 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe46bfc2-36dd-491f-9b6b-5b7bbea27086","Type":"ContainerStarted","Data":"25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8"} Jul 14 18:23:22.792502 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:22.792111 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe46bfc2-36dd-491f-9b6b-5b7bbea27086","Type":"ContainerStarted","Data":"b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb"} Jul 14 18:23:22.792502 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:22.792150 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe46bfc2-36dd-491f-9b6b-5b7bbea27086","Type":"ContainerStarted","Data":"5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09"} Jul 14 18:23:22.860271 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:22.860205 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=10.361485724 podStartE2EDuration="15.860180447s" totalImagesPullingTime="5.473575036s" totalInitContainerRuntime="25.119687ms" isStatefulPod=false podCreationTimestamp="2026-07-14 18:23:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:22.858688653 +0000 UTC m=+104.396136439" watchObservedRunningTime="2026-07-14 18:23:22.860180447 +0000 UTC m=+104.397628232" Jul 14 18:23:25.813888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:25.813829 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:25.896414 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:25.896380 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:27.798777 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:27.798748 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-797996db74-bzk2c" Jul 14 18:23:38.711427 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:38.711391 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:38.796963 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:38.796930 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:41.808747 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:41.808712 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:41.872626 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:41.872593 2573 generic.go:356] "Generic (PLEG): container finished" podID="6392abcd-2ccd-410b-8c10-08a58197237e" containerID="d50c5b15c0501e9da6670186c91b502b55545c041bfbe71a69976685aaf42431" exitCode=0 Jul 14 18:23:41.872758 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:41.872665 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" event={"ID":"6392abcd-2ccd-410b-8c10-08a58197237e","Type":"ContainerDied","Data":"d50c5b15c0501e9da6670186c91b502b55545c041bfbe71a69976685aaf42431"} Jul 14 18:23:41.873014 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:41.873002 2573 scope.go:117] "RemoveContainer" containerID="d50c5b15c0501e9da6670186c91b502b55545c041bfbe71a69976685aaf42431" Jul 14 18:23:41.917277 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:41.917254 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:42.878646 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:42.878612 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-tkrhd" event={"ID":"6392abcd-2ccd-410b-8c10-08a58197237e","Type":"ContainerStarted","Data":"eadd91f2d9b00417ad3b970fe30278e8ee9a371aecf0e58154baa8a9bee9acf3"} Jul 14 18:23:44.921305 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:44.921229 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" podUID="4e730311-d011-495f-a4c7-a0097413011c" containerName="registry" containerID="cri-o://ae26cefa60abc3c46c85366750a23230186dbfab6781d82d6742f6f2fd72b042" gracePeriod=30 Jul 14 18:23:45.203091 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.203064 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:23:45.308794 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.308766 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-bound-sa-token\") pod \"4e730311-d011-495f-a4c7-a0097413011c\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " Jul 14 18:23:45.308946 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.308803 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e730311-d011-495f-a4c7-a0097413011c-installation-pull-secrets\") pod \"4e730311-d011-495f-a4c7-a0097413011c\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " Jul 14 18:23:45.308946 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.308834 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e730311-d011-495f-a4c7-a0097413011c-trusted-ca\") pod \"4e730311-d011-495f-a4c7-a0097413011c\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " Jul 14 18:23:45.308946 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.308865 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls\") pod \"4e730311-d011-495f-a4c7-a0097413011c\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " Jul 14 18:23:45.308946 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.308896 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e730311-d011-495f-a4c7-a0097413011c-ca-trust-extracted\") pod \"4e730311-d011-495f-a4c7-a0097413011c\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " Jul 14 18:23:45.309152 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.308979 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e730311-d011-495f-a4c7-a0097413011c-registry-certificates\") pod \"4e730311-d011-495f-a4c7-a0097413011c\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " Jul 14 18:23:45.309152 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.309006 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4lq7\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-kube-api-access-r4lq7\") pod \"4e730311-d011-495f-a4c7-a0097413011c\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " Jul 14 18:23:45.309152 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.309046 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e730311-d011-495f-a4c7-a0097413011c-image-registry-private-configuration\") pod \"4e730311-d011-495f-a4c7-a0097413011c\" (UID: \"4e730311-d011-495f-a4c7-a0097413011c\") " Jul 14 18:23:45.309619 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.309338 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e730311-d011-495f-a4c7-a0097413011c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4e730311-d011-495f-a4c7-a0097413011c" (UID: "4e730311-d011-495f-a4c7-a0097413011c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:23:45.309619 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.309576 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e730311-d011-495f-a4c7-a0097413011c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4e730311-d011-495f-a4c7-a0097413011c" (UID: "4e730311-d011-495f-a4c7-a0097413011c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:23:45.311709 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.311652 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4e730311-d011-495f-a4c7-a0097413011c" (UID: "4e730311-d011-495f-a4c7-a0097413011c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:23:45.311709 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.311677 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e730311-d011-495f-a4c7-a0097413011c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4e730311-d011-495f-a4c7-a0097413011c" (UID: "4e730311-d011-495f-a4c7-a0097413011c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:23:45.311863 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.311736 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-kube-api-access-r4lq7" (OuterVolumeSpecName: "kube-api-access-r4lq7") pod "4e730311-d011-495f-a4c7-a0097413011c" (UID: "4e730311-d011-495f-a4c7-a0097413011c"). InnerVolumeSpecName "kube-api-access-r4lq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:23:45.311863 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.311744 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e730311-d011-495f-a4c7-a0097413011c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4e730311-d011-495f-a4c7-a0097413011c" (UID: "4e730311-d011-495f-a4c7-a0097413011c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:23:45.311863 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.311821 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4e730311-d011-495f-a4c7-a0097413011c" (UID: "4e730311-d011-495f-a4c7-a0097413011c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:23:45.318340 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.318265 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e730311-d011-495f-a4c7-a0097413011c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4e730311-d011-495f-a4c7-a0097413011c" (UID: "4e730311-d011-495f-a4c7-a0097413011c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:23:45.409688 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.409664 2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-registry-tls\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:23:45.409688 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.409686 2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e730311-d011-495f-a4c7-a0097413011c-ca-trust-extracted\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:23:45.409825 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.409697 2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e730311-d011-495f-a4c7-a0097413011c-registry-certificates\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:23:45.409825 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.409706 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4lq7\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-kube-api-access-r4lq7\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:23:45.409825 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.409716 2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e730311-d011-495f-a4c7-a0097413011c-image-registry-private-configuration\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:23:45.409825 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.409726 2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e730311-d011-495f-a4c7-a0097413011c-bound-sa-token\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:23:45.409825 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.409734 2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e730311-d011-495f-a4c7-a0097413011c-installation-pull-secrets\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:23:45.409825 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.409742 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e730311-d011-495f-a4c7-a0097413011c-trusted-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:23:45.892496 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.892467 2573 generic.go:356] "Generic (PLEG): container finished" podID="4e730311-d011-495f-a4c7-a0097413011c" containerID="ae26cefa60abc3c46c85366750a23230186dbfab6781d82d6742f6f2fd72b042" exitCode=0 Jul 14 18:23:45.892683 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.892517 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" event={"ID":"4e730311-d011-495f-a4c7-a0097413011c","Type":"ContainerDied","Data":"ae26cefa60abc3c46c85366750a23230186dbfab6781d82d6742f6f2fd72b042"} Jul 14 18:23:45.892683 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.892525 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" Jul 14 18:23:45.892683 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.892558 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68877f6bf8-lg2z8" event={"ID":"4e730311-d011-495f-a4c7-a0097413011c","Type":"ContainerDied","Data":"5bb421857cb1d54794f198e869bb2bca14009f996875f1cdecb61891520e3b9f"} Jul 14 18:23:45.892683 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.892579 2573 scope.go:117] "RemoveContainer" containerID="ae26cefa60abc3c46c85366750a23230186dbfab6781d82d6742f6f2fd72b042" Jul 14 18:23:45.901230 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.901211 2573 scope.go:117] "RemoveContainer" containerID="ae26cefa60abc3c46c85366750a23230186dbfab6781d82d6742f6f2fd72b042" Jul 14 18:23:45.901574 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:23:45.901554 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae26cefa60abc3c46c85366750a23230186dbfab6781d82d6742f6f2fd72b042\": container with ID starting with ae26cefa60abc3c46c85366750a23230186dbfab6781d82d6742f6f2fd72b042 not found: ID does not exist" containerID="ae26cefa60abc3c46c85366750a23230186dbfab6781d82d6742f6f2fd72b042" Jul 14 18:23:45.901635 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.901586 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae26cefa60abc3c46c85366750a23230186dbfab6781d82d6742f6f2fd72b042"} err="failed to get container status \"ae26cefa60abc3c46c85366750a23230186dbfab6781d82d6742f6f2fd72b042\": rpc error: code = NotFound desc = could not find container \"ae26cefa60abc3c46c85366750a23230186dbfab6781d82d6742f6f2fd72b042\": container with ID starting with ae26cefa60abc3c46c85366750a23230186dbfab6781d82d6742f6f2fd72b042 not found: ID does not exist" Jul 14 18:23:45.919658 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.919635 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68877f6bf8-lg2z8"] Jul 14 18:23:45.926012 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:45.925981 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68877f6bf8-lg2z8"] Jul 14 18:23:47.221074 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:47.221042 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e730311-d011-495f-a4c7-a0097413011c" path="/var/lib/kubelet/pods/4e730311-d011-495f-a4c7-a0097413011c/volumes" Jul 14 18:23:55.302806 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:55.302710 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:55.404001 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:55.403953 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:23:56.951341 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:56.951279 2573 generic.go:356] "Generic (PLEG): container finished" podID="42555e5b-8ae2-4677-a74a-bbacb3393030" containerID="c3c563e837e6f31646c6f91c8e2634ebae33ac15e51ed15d7c38b9441b399f05" exitCode=0 Jul 14 18:23:56.951684 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:56.951331 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" event={"ID":"42555e5b-8ae2-4677-a74a-bbacb3393030","Type":"ContainerDied","Data":"c3c563e837e6f31646c6f91c8e2634ebae33ac15e51ed15d7c38b9441b399f05"} Jul 14 18:23:56.951684 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:56.951648 2573 scope.go:117] "RemoveContainer" containerID="c3c563e837e6f31646c6f91c8e2634ebae33ac15e51ed15d7c38b9441b399f05" Jul 14 18:23:57.958625 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:57.958589 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-n22g8" event={"ID":"42555e5b-8ae2-4677-a74a-bbacb3393030","Type":"ContainerStarted","Data":"2d28ba70e03b67a4579f8e81ba055134614342db144b01d8d177c8277e250325"} Jul 14 18:23:57.960054 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:57.960030 2573 generic.go:356] "Generic (PLEG): container finished" podID="113b1fbe-22b0-4d6f-8f4f-ede50c3b9242" containerID="4e52a1b71cc3d2f2866eddb4aaa79752ba18b51f3862dba3ec83e6a0210bcf69" exitCode=0 Jul 14 18:23:57.960156 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:57.960075 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" event={"ID":"113b1fbe-22b0-4d6f-8f4f-ede50c3b9242","Type":"ContainerDied","Data":"4e52a1b71cc3d2f2866eddb4aaa79752ba18b51f3862dba3ec83e6a0210bcf69"} Jul 14 18:23:57.960334 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:57.960322 2573 scope.go:117] "RemoveContainer" containerID="4e52a1b71cc3d2f2866eddb4aaa79752ba18b51f3862dba3ec83e6a0210bcf69" Jul 14 18:23:58.967026 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:23:58.966990 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-m94bg" event={"ID":"113b1fbe-22b0-4d6f-8f4f-ede50c3b9242","Type":"ContainerStarted","Data":"51306c9d375c2a0b4385759fff6d65adc146c038aa00053af56ddf7f1dc28f2e"} Jul 14 18:24:25.450624 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:25.450583 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:24:25.654817 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:25.654779 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:24:26.531261 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:26.531234 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 18:24:26.531655 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:26.531522 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="alertmanager" containerID="cri-o://4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292" gracePeriod=120 Jul 14 18:24:26.531655 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:26.531584 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="kube-rbac-proxy-metric" containerID="cri-o://25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8" gracePeriod=120 Jul 14 18:24:26.531655 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:26.531604 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="kube-rbac-proxy-web" containerID="cri-o://5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09" gracePeriod=120 Jul 14 18:24:26.531826 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:26.531647 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="config-reloader" containerID="cri-o://d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850" gracePeriod=120 Jul 14 18:24:26.531826 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:26.531612 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="kube-rbac-proxy" containerID="cri-o://b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb" gracePeriod=120 Jul 14 18:24:26.531826 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:26.531663 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="prom-label-proxy" containerID="cri-o://4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61" gracePeriod=120 Jul 14 18:24:27.085947 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.085918 2573 generic.go:356] "Generic (PLEG): container finished" podID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerID="4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61" exitCode=0 Jul 14 18:24:27.085947 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.085941 2573 generic.go:356] "Generic (PLEG): container finished" podID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerID="b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb" exitCode=0 Jul 14 18:24:27.085947 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.085947 2573 generic.go:356] "Generic (PLEG): container finished" podID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerID="d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850" exitCode=0 Jul 14 18:24:27.085947 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.085953 2573 generic.go:356] "Generic (PLEG): container finished" podID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerID="4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292" exitCode=0 Jul 14 18:24:27.086206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.085989 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe46bfc2-36dd-491f-9b6b-5b7bbea27086","Type":"ContainerDied","Data":"4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61"} Jul 14 18:24:27.086206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.086080 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe46bfc2-36dd-491f-9b6b-5b7bbea27086","Type":"ContainerDied","Data":"b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb"} Jul 14 18:24:27.086206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.086109 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe46bfc2-36dd-491f-9b6b-5b7bbea27086","Type":"ContainerDied","Data":"d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850"} Jul 14 18:24:27.086206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.086134 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe46bfc2-36dd-491f-9b6b-5b7bbea27086","Type":"ContainerDied","Data":"4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292"} Jul 14 18:24:27.777831 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.777809 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:27.955362 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.955331 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy-web\") pod \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " Jul 14 18:24:27.955533 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.955370 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-cluster-tls-config\") pod \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " Jul 14 18:24:27.955533 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.955403 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy\") pod \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " Jul 14 18:24:27.955533 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.955420 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-metrics-client-ca\") pod \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " Jul 14 18:24:27.955533 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.955447 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pjnp\" (UniqueName: \"kubernetes.io/projected/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-kube-api-access-5pjnp\") pod \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " Jul 14 18:24:27.955533 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.955474 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-web-config\") pod \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " Jul 14 18:24:27.955533 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.955494 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-alertmanager-main-db\") pod \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " Jul 14 18:24:27.955533 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.955510 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy-metric\") pod \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " Jul 14 18:24:27.955874 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.955551 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-config-volume\") pod \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " Jul 14 18:24:27.955874 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.955579 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-main-tls\") pod \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " Jul 14 18:24:27.955874 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.955616 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-tls-assets\") pod \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " Jul 14 18:24:27.955874 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.955670 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-alertmanager-trusted-ca-bundle\") pod \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " Jul 14 18:24:27.955874 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.955738 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-config-out\") pod \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\" (UID: \"fe46bfc2-36dd-491f-9b6b-5b7bbea27086\") " Jul 14 18:24:27.955874 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.955815 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "fe46bfc2-36dd-491f-9b6b-5b7bbea27086" (UID: "fe46bfc2-36dd-491f-9b6b-5b7bbea27086"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:24:27.956170 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.955982 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-metrics-client-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:24:27.956170 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.956154 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "fe46bfc2-36dd-491f-9b6b-5b7bbea27086" (UID: "fe46bfc2-36dd-491f-9b6b-5b7bbea27086"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:24:27.956451 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.956418 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "fe46bfc2-36dd-491f-9b6b-5b7bbea27086" (UID: "fe46bfc2-36dd-491f-9b6b-5b7bbea27086"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:24:27.958079 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.958055 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "fe46bfc2-36dd-491f-9b6b-5b7bbea27086" (UID: "fe46bfc2-36dd-491f-9b6b-5b7bbea27086"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:27.958683 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.958639 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "fe46bfc2-36dd-491f-9b6b-5b7bbea27086" (UID: "fe46bfc2-36dd-491f-9b6b-5b7bbea27086"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:27.958870 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.958838 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "fe46bfc2-36dd-491f-9b6b-5b7bbea27086" (UID: "fe46bfc2-36dd-491f-9b6b-5b7bbea27086"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:27.958964 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.958899 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-kube-api-access-5pjnp" (OuterVolumeSpecName: "kube-api-access-5pjnp") pod "fe46bfc2-36dd-491f-9b6b-5b7bbea27086" (UID: "fe46bfc2-36dd-491f-9b6b-5b7bbea27086"). InnerVolumeSpecName "kube-api-access-5pjnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:24:27.959258 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.959238 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-config-out" (OuterVolumeSpecName: "config-out") pod "fe46bfc2-36dd-491f-9b6b-5b7bbea27086" (UID: "fe46bfc2-36dd-491f-9b6b-5b7bbea27086"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:24:27.959768 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.959744 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fe46bfc2-36dd-491f-9b6b-5b7bbea27086" (UID: "fe46bfc2-36dd-491f-9b6b-5b7bbea27086"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:24:27.960159 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.960139 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "fe46bfc2-36dd-491f-9b6b-5b7bbea27086" (UID: "fe46bfc2-36dd-491f-9b6b-5b7bbea27086"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:27.960218 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.960200 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-config-volume" (OuterVolumeSpecName: "config-volume") pod "fe46bfc2-36dd-491f-9b6b-5b7bbea27086" (UID: "fe46bfc2-36dd-491f-9b6b-5b7bbea27086"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:27.963055 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.963021 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "fe46bfc2-36dd-491f-9b6b-5b7bbea27086" (UID: "fe46bfc2-36dd-491f-9b6b-5b7bbea27086"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:27.970185 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:27.970160 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-web-config" (OuterVolumeSpecName: "web-config") pod "fe46bfc2-36dd-491f-9b6b-5b7bbea27086" (UID: "fe46bfc2-36dd-491f-9b6b-5b7bbea27086"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:28.057385 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.057350 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pjnp\" (UniqueName: \"kubernetes.io/projected/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-kube-api-access-5pjnp\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:24:28.057385 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.057381 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-web-config\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:24:28.057385 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.057392 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-alertmanager-main-db\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:24:28.057597 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.057402 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:24:28.057597 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.057413 2573 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-config-volume\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:24:28.057597 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.057421 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-main-tls\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:24:28.057597 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.057428 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-tls-assets\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:24:28.057597 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.057438 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:24:28.057597 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.057448 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-config-out\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:24:28.057597 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.057458 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:24:28.057597 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.057466 2573 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-cluster-tls-config\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:24:28.057597 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.057475 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe46bfc2-36dd-491f-9b6b-5b7bbea27086-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:24:28.092862 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.092833 2573 generic.go:356] "Generic (PLEG): container finished" podID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerID="25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8" exitCode=0 Jul 14 18:24:28.092862 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.092857 2573 generic.go:356] "Generic (PLEG): container finished" podID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerID="5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09" exitCode=0 Jul 14 18:24:28.093048 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.092914 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.093048 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.092925 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe46bfc2-36dd-491f-9b6b-5b7bbea27086","Type":"ContainerDied","Data":"25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8"} Jul 14 18:24:28.093048 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.092998 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe46bfc2-36dd-491f-9b6b-5b7bbea27086","Type":"ContainerDied","Data":"5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09"} Jul 14 18:24:28.093048 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.093026 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe46bfc2-36dd-491f-9b6b-5b7bbea27086","Type":"ContainerDied","Data":"6866e570e5eb20b0a0ac323c1061a43dcdcb8041bb74ecc7c8d33b5b7ee6d08a"} Jul 14 18:24:28.093163 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.093057 2573 scope.go:117] "RemoveContainer" containerID="4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61" Jul 14 18:24:28.101742 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.101727 2573 scope.go:117] "RemoveContainer" containerID="25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8" Jul 14 18:24:28.110824 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.110807 2573 scope.go:117] "RemoveContainer" containerID="b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb" Jul 14 18:24:28.117780 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.117759 2573 scope.go:117] "RemoveContainer" containerID="5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09" Jul 14 18:24:28.121488 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.121469 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 18:24:28.124867 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.124849 2573 scope.go:117] "RemoveContainer" containerID="d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850" Jul 14 18:24:28.128197 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.128173 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 18:24:28.132158 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.132141 2573 scope.go:117] "RemoveContainer" containerID="4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292" Jul 14 18:24:28.138885 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.138869 2573 scope.go:117] "RemoveContainer" containerID="8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e" Jul 14 18:24:28.145471 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.145457 2573 scope.go:117] "RemoveContainer" containerID="4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61" Jul 14 18:24:28.145731 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:24:28.145712 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61\": container with ID starting with 4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61 not found: ID does not exist" containerID="4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61" Jul 14 18:24:28.145798 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.145741 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61"} err="failed to get container status \"4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61\": rpc error: code = NotFound desc = could not find container \"4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61\": container with ID starting with 4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61 not found: ID does not exist" Jul 14 18:24:28.145798 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.145760 2573 scope.go:117] "RemoveContainer" containerID="25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8" Jul 14 18:24:28.146001 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:24:28.145984 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8\": container with ID starting with 25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8 not found: ID does not exist" containerID="25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8" Jul 14 18:24:28.146039 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.146011 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8"} err="failed to get container status \"25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8\": rpc error: code = NotFound desc = could not find container \"25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8\": container with ID starting with 25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8 not found: ID does not exist" Jul 14 18:24:28.146039 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.146029 2573 scope.go:117] "RemoveContainer" containerID="b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb" Jul 14 18:24:28.146269 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:24:28.146251 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb\": container with ID starting with b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb not found: ID does not exist" containerID="b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb" Jul 14 18:24:28.146351 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.146273 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb"} err="failed to get container status \"b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb\": rpc error: code = NotFound desc = could not find container \"b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb\": container with ID starting with b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb not found: ID does not exist" Jul 14 18:24:28.146351 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.146305 2573 scope.go:117] "RemoveContainer" containerID="5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09" Jul 14 18:24:28.146535 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:24:28.146517 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09\": container with ID starting with 5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09 not found: ID does not exist" containerID="5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09" Jul 14 18:24:28.146579 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.146539 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09"} err="failed to get container status \"5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09\": rpc error: code = NotFound desc = could not find container \"5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09\": container with ID starting with 5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09 not found: ID does not exist" Jul 14 18:24:28.146579 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.146554 2573 scope.go:117] "RemoveContainer" containerID="d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850" Jul 14 18:24:28.146786 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:24:28.146768 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850\": container with ID starting with d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850 not found: ID does not exist" containerID="d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850" Jul 14 18:24:28.146845 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.146796 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850"} err="failed to get container status \"d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850\": rpc error: code = NotFound desc = could not find container \"d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850\": container with ID starting with d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850 not found: ID does not exist" Jul 14 18:24:28.146845 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.146817 2573 scope.go:117] "RemoveContainer" containerID="4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292" Jul 14 18:24:28.147066 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:24:28.147049 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292\": container with ID starting with 4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292 not found: ID does not exist" containerID="4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292" Jul 14 18:24:28.147103 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.147069 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292"} err="failed to get container status \"4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292\": rpc error: code = NotFound desc = could not find container \"4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292\": container with ID starting with 4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292 not found: ID does not exist" Jul 14 18:24:28.147103 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.147083 2573 scope.go:117] "RemoveContainer" containerID="8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e" Jul 14 18:24:28.147362 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:24:28.147314 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e\": container with ID starting with 8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e not found: ID does not exist" containerID="8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e" Jul 14 18:24:28.147362 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.147342 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e"} err="failed to get container status \"8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e\": rpc error: code = NotFound desc = could not find container \"8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e\": container with ID starting with 8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e not found: ID does not exist" Jul 14 18:24:28.147362 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.147358 2573 scope.go:117] "RemoveContainer" containerID="4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61" Jul 14 18:24:28.147604 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.147587 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61"} err="failed to get container status \"4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61\": rpc error: code = NotFound desc = could not find container \"4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61\": container with ID starting with 4186c3de6ce4198c017151a7a227ba5d80685345ddbbe2aff546f6608ba5ac61 not found: ID does not exist" Jul 14 18:24:28.147604 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.147605 2573 scope.go:117] "RemoveContainer" containerID="25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8" Jul 14 18:24:28.147827 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.147808 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8"} err="failed to get container status \"25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8\": rpc error: code = NotFound desc = could not find container \"25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8\": container with ID starting with 25d97a406408b3311d93881b82ca3501d34c570171df418344a6e7163c51f1b8 not found: ID does not exist" Jul 14 18:24:28.147883 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.147828 2573 scope.go:117] "RemoveContainer" containerID="b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb" Jul 14 18:24:28.148042 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.148027 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb"} err="failed to get container status \"b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb\": rpc error: code = NotFound desc = could not find container \"b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb\": container with ID starting with b4daad4e6f6c067a7c497cc3a1e21852586661e8cf66a0874e9f4c586ca7d8bb not found: ID does not exist" Jul 14 18:24:28.148086 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.148042 2573 scope.go:117] "RemoveContainer" containerID="5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09" Jul 14 18:24:28.148222 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.148206 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09"} err="failed to get container status \"5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09\": rpc error: code = NotFound desc = could not find container \"5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09\": container with ID starting with 5e92c0573421160c954e78a11eca05b6c2064d613c9957fec6af100f42e1ae09 not found: ID does not exist" Jul 14 18:24:28.148258 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.148221 2573 scope.go:117] "RemoveContainer" containerID="d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850" Jul 14 18:24:28.148465 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.148441 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850"} err="failed to get container status \"d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850\": rpc error: code = NotFound desc = could not find container \"d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850\": container with ID starting with d6d2832bde58e87dc998302e50e58af4d8c2577be963bce41c45dc0f650d4850 not found: ID does not exist" Jul 14 18:24:28.148524 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.148466 2573 scope.go:117] "RemoveContainer" containerID="4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292" Jul 14 18:24:28.148691 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.148675 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292"} err="failed to get container status \"4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292\": rpc error: code = NotFound desc = could not find container \"4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292\": container with ID starting with 4e84809c0a8078999eaebd04af20a6577cb47967debca4722abad1dcbd52a292 not found: ID does not exist" Jul 14 18:24:28.148779 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.148690 2573 scope.go:117] "RemoveContainer" containerID="8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e" Jul 14 18:24:28.148888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.148873 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e"} err="failed to get container status \"8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e\": rpc error: code = NotFound desc = could not find container \"8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e\": container with ID starting with 8ba6c3d62acc1f105a95ca327f45135deb49deff0b8f597ca0d5e384cc6c4b6e not found: ID does not exist" Jul 14 18:24:28.161875 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.161851 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 18:24:28.162141 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162130 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="config-reloader" Jul 14 18:24:28.162182 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162143 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="config-reloader" Jul 14 18:24:28.162182 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162153 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="alertmanager" Jul 14 18:24:28.162182 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162158 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="alertmanager" Jul 14 18:24:28.162182 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162166 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="prom-label-proxy" Jul 14 18:24:28.162182 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162172 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="prom-label-proxy" Jul 14 18:24:28.162182 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162182 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="init-config-reloader" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162187 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="init-config-reloader" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162194 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e730311-d011-495f-a4c7-a0097413011c" containerName="registry" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162199 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e730311-d011-495f-a4c7-a0097413011c" containerName="registry" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162207 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="kube-rbac-proxy-web" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162212 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="kube-rbac-proxy-web" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162226 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="kube-rbac-proxy" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162230 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="kube-rbac-proxy" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162236 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="kube-rbac-proxy-metric" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162241 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="kube-rbac-proxy-metric" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162309 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="alertmanager" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162318 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="config-reloader" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162324 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="kube-rbac-proxy-web" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162330 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="kube-rbac-proxy-metric" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162337 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="kube-rbac-proxy" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162344 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" containerName="prom-label-proxy" Jul 14 18:24:28.162370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.162349 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e730311-d011-495f-a4c7-a0097413011c" containerName="registry" Jul 14 18:24:28.167187 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.167169 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.170395 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.170374 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 14 18:24:28.170395 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.170388 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 14 18:24:28.170500 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.170390 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 14 18:24:28.170728 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.170712 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 14 18:24:28.170854 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.170791 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 14 18:24:28.170854 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.170818 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 14 18:24:28.171018 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.170899 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c5l48\"" Jul 14 18:24:28.171018 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.170954 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 14 18:24:28.171143 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.171026 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 14 18:24:28.178492 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.178464 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 14 18:24:28.195604 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.195581 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 18:24:28.360306 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.360213 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.360306 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.360248 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.360534 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.360378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.360534 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.360418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5324143-4949-4048-aba8-6c9e979ab8dc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.360534 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.360450 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.360534 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.360492 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2mrb\" (UniqueName: \"kubernetes.io/projected/d5324143-4949-4048-aba8-6c9e979ab8dc-kube-api-access-g2mrb\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.360667 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.360559 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5324143-4949-4048-aba8-6c9e979ab8dc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.360667 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.360598 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.360667 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.360616 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5324143-4949-4048-aba8-6c9e979ab8dc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.360667 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.360632 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5324143-4949-4048-aba8-6c9e979ab8dc-config-out\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.360667 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.360655 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-web-config\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.360823 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.360675 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5324143-4949-4048-aba8-6c9e979ab8dc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.360823 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.360719 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.462096 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.462067 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.462262 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.462099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.462262 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.462129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.462262 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.462157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5324143-4949-4048-aba8-6c9e979ab8dc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.462439 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.462337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.462439 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.462375 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2mrb\" (UniqueName: \"kubernetes.io/projected/d5324143-4949-4048-aba8-6c9e979ab8dc-kube-api-access-g2mrb\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.462578 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.462538 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5324143-4949-4048-aba8-6c9e979ab8dc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.462660 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.462600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.462660 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.462629 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5324143-4949-4048-aba8-6c9e979ab8dc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.462660 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.462656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5324143-4949-4048-aba8-6c9e979ab8dc-config-out\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.462819 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.462684 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-web-config\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.463161 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.463129 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5324143-4949-4048-aba8-6c9e979ab8dc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.463828 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.463349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5324143-4949-4048-aba8-6c9e979ab8dc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.463828 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.463444 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.463828 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.463771 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5324143-4949-4048-aba8-6c9e979ab8dc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.464059 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.464028 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5324143-4949-4048-aba8-6c9e979ab8dc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.465194 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.465138 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.465309 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.465222 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.465439 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.465417 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5324143-4949-4048-aba8-6c9e979ab8dc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.465525 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.465504 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.465714 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.465695 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.465814 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.465801 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5324143-4949-4048-aba8-6c9e979ab8dc-config-out\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.466164 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.466144 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.466631 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.466616 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-web-config\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.466683 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.466665 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5324143-4949-4048-aba8-6c9e979ab8dc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.473852 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.473836 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2mrb\" (UniqueName: \"kubernetes.io/projected/d5324143-4949-4048-aba8-6c9e979ab8dc-kube-api-access-g2mrb\") pod \"alertmanager-main-0\" (UID: \"d5324143-4949-4048-aba8-6c9e979ab8dc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.477712 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.477697 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 14 18:24:28.635060 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:28.635022 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 14 18:24:28.638881 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:24:28.638855 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5324143_4949_4048_aba8_6c9e979ab8dc.slice/crio-9042725cc8e0653c5415e004d9538447113dea28c2c6f73fa4d655631cb0af4c WatchSource:0}: Error finding container 9042725cc8e0653c5415e004d9538447113dea28c2c6f73fa4d655631cb0af4c: Status 404 returned error can't find the container with id 9042725cc8e0653c5415e004d9538447113dea28c2c6f73fa4d655631cb0af4c Jul 14 18:24:29.099516 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:29.099481 2573 generic.go:356] "Generic (PLEG): container finished" podID="d5324143-4949-4048-aba8-6c9e979ab8dc" containerID="6bb4c1a1be53ab59d768cc13fc7ea0228781519593bd1aec6c2405024cbfe22b" exitCode=0 Jul 14 18:24:29.099855 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:29.099522 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5324143-4949-4048-aba8-6c9e979ab8dc","Type":"ContainerDied","Data":"6bb4c1a1be53ab59d768cc13fc7ea0228781519593bd1aec6c2405024cbfe22b"} Jul 14 18:24:29.099855 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:29.099593 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5324143-4949-4048-aba8-6c9e979ab8dc","Type":"ContainerStarted","Data":"9042725cc8e0653c5415e004d9538447113dea28c2c6f73fa4d655631cb0af4c"} Jul 14 18:24:29.222493 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:29.222470 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe46bfc2-36dd-491f-9b6b-5b7bbea27086" path="/var/lib/kubelet/pods/fe46bfc2-36dd-491f-9b6b-5b7bbea27086/volumes" Jul 14 18:24:30.107605 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.107566 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5324143-4949-4048-aba8-6c9e979ab8dc","Type":"ContainerStarted","Data":"b9122ad4fbfd62910e25c2e5f93f11faae1643e47d5c10c36fef9a372b1ac44f"} Jul 14 18:24:30.108047 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.107643 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5324143-4949-4048-aba8-6c9e979ab8dc","Type":"ContainerStarted","Data":"eb33d2b9b09de2acc0a9ca00e8291a9c3f1b39ecd22d3de289e90b94690afa14"} Jul 14 18:24:30.108047 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.107682 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5324143-4949-4048-aba8-6c9e979ab8dc","Type":"ContainerStarted","Data":"78c78cc8cc42496acb0e96551c7340c78325d1ab8b94e7830ca638ea679bc0a7"} Jul 14 18:24:30.108047 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.107729 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5324143-4949-4048-aba8-6c9e979ab8dc","Type":"ContainerStarted","Data":"0e4adf60b192534f06cadc44babdd558ef88a1972fe0939fee028d346e13b964"} Jul 14 18:24:30.108047 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.107770 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5324143-4949-4048-aba8-6c9e979ab8dc","Type":"ContainerStarted","Data":"ef2c44388bc55a8d7edc89e9c4a0aa0c5667ffb080fd4bf7a84624d2abcf1e90"} Jul 14 18:24:30.108047 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.107812 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5324143-4949-4048-aba8-6c9e979ab8dc","Type":"ContainerStarted","Data":"e85f664e3ba859c21ca72a245b6d2f18968af37a47194b57950a18dafce63359"} Jul 14 18:24:30.143369 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.143314 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.125949978 podStartE2EDuration="2.143298281s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.348303ms" isStatefulPod=false podCreationTimestamp="2026-07-14 18:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:24:30.140489702 +0000 UTC m=+171.677937488" watchObservedRunningTime="2026-07-14 18:24:30.143298281 +0000 UTC m=+171.680746099" Jul 14 18:24:30.573013 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.572983 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6b665d5f69-xbh9c"] Jul 14 18:24:30.576242 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.576227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.579244 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.579213 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 14 18:24:30.579244 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.579221 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 14 18:24:30.579657 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.579265 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 14 18:24:30.579657 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.579303 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 14 18:24:30.579657 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.579591 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xrnfw\"" Jul 14 18:24:30.579768 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.579688 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 14 18:24:30.580830 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.580807 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-serving-certs-ca-bundle\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.580960 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.580846 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.580960 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.580873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-telemeter-client-tls\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.581069 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.580956 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-secret-telemeter-client\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.581069 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.581027 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8lxj\" (UniqueName: \"kubernetes.io/projected/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-kube-api-access-d8lxj\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.581173 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.581091 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-federate-client-tls\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.581173 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.581160 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-metrics-client-ca\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.581274 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.581196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.587216 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.587185 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 14 18:24:30.594014 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.593991 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b665d5f69-xbh9c"] Jul 14 18:24:30.682144 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.682085 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8lxj\" (UniqueName: \"kubernetes.io/projected/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-kube-api-access-d8lxj\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.682144 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.682118 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-federate-client-tls\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.682308 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.682147 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-metrics-client-ca\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.682308 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.682169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.682308 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.682222 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-serving-certs-ca-bundle\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.682308 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.682249 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.682308 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.682277 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-telemeter-client-tls\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.682542 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.682487 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-secret-telemeter-client\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.683179 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.683154 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-serving-certs-ca-bundle\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.683359 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.683157 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-metrics-client-ca\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.683461 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.683439 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.684891 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.684866 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.685038 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.685017 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-secret-telemeter-client\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.685096 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.685081 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-telemeter-client-tls\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.685134 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.685121 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-federate-client-tls\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.692266 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.692238 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8lxj\" (UniqueName: \"kubernetes.io/projected/0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb-kube-api-access-d8lxj\") pod \"telemeter-client-6b665d5f69-xbh9c\" (UID: \"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb\") " pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:30.892251 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:30.892173 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" Jul 14 18:24:31.034410 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:31.034385 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b665d5f69-xbh9c"] Jul 14 18:24:31.035242 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:24:31.035217 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b11cad7_ebf8_44ed_a0db_bd6b36b5e2cb.slice/crio-46d7e5e3b36472b98638299ef96f8214c4e1cc8e59dee96f8fa8e63d64fd7fce WatchSource:0}: Error finding container 46d7e5e3b36472b98638299ef96f8214c4e1cc8e59dee96f8fa8e63d64fd7fce: Status 404 returned error can't find the container with id 46d7e5e3b36472b98638299ef96f8214c4e1cc8e59dee96f8fa8e63d64fd7fce Jul 14 18:24:31.113463 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:31.113435 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" event={"ID":"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb","Type":"ContainerStarted","Data":"46d7e5e3b36472b98638299ef96f8214c4e1cc8e59dee96f8fa8e63d64fd7fce"} Jul 14 18:24:34.131406 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:34.131368 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" event={"ID":"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb","Type":"ContainerStarted","Data":"0c20cdde83492ac955ecd1bdb8e4da876907ac6b5df42478845b9b0ffa795975"} Jul 14 18:24:34.131864 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:34.131434 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" event={"ID":"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb","Type":"ContainerStarted","Data":"4e20abf6fc3f9c13ffa5d68901cdb5740e9bb8eebffe8841c19faa23dd08cae5"} Jul 14 18:24:34.131864 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:34.131462 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" event={"ID":"0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb","Type":"ContainerStarted","Data":"06441fcdeed100ac59ddd0ca91f68e5a02cdf5d6e99c0d86d058b11013033d91"} Jul 14 18:24:34.161611 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:34.161566 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6b665d5f69-xbh9c" podStartSLOduration=1.539370805 podStartE2EDuration="4.161549884s" totalImagesPullingTime="2.622179079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:24:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:24:34.160201287 +0000 UTC m=+175.697649072" watchObservedRunningTime="2026-07-14 18:24:34.161549884 +0000 UTC m=+175.698997667" Jul 14 18:24:35.036486 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:35.036449 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:24:35.305449 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:35.305371 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:24:41.670378 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:41.670346 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:24:42.847067 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:42.847025 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:24:46.424240 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:46.424208 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:24:46.535310 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:24:46.535260 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:25:08.033473 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:25:08.033437 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:25:12.919607 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:25:12.919570 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:25:14.119539 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:25:14.119502 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:25:21.438914 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:25:21.438880 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:25:40.809457 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:25:40.809424 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:25:40.905929 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:25:40.905899 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:25:52.701649 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:25:52.701611 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:25:52.866398 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:25:52.866368 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:25:55.175751 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:25:55.175712 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:25:55.324218 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:25:55.324185 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:26:05.468728 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:26:05.468696 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:26:05.747524 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:26:05.747442 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:26:05.848193 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:26:05.848156 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:26:08.760507 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:26:08.760474 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:26:08.877214 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:26:08.877185 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:26:34.631533 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:26:34.631500 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:26:34.779410 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:26:34.779379 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:26:38.802378 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:26:38.802340 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:26:38.804722 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:26:38.804696 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:26:38.810617 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:26:38.810589 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:26:38.812391 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:26:38.812374 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:26:38.812638 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:26:38.812621 2573 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 18:27:01.695010 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:01.694947 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:03.074556 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:03.074515 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:05.689269 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:05.689232 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:05.807525 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:05.807489 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:07.230015 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:07.229986 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:07.378466 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:07.378433 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:08.090231 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:08.090185 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:08.244428 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:08.244379 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:08.606655 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:08.606620 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jr6b9"] Jul 14 18:27:08.609820 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:08.609802 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jr6b9" Jul 14 18:27:08.615543 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:08.615521 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 14 18:27:08.615636 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:08.615528 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 14 18:27:08.616709 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:08.616693 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-b9csh\"" Jul 14 18:27:08.633392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:08.633366 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jr6b9"] Jul 14 18:27:08.785062 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:08.785032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81021233-9462-48e8-9cb2-b4165344a604-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jr6b9\" (UID: \"81021233-9462-48e8-9cb2-b4165344a604\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr6b9" Jul 14 18:27:08.785244 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:08.785089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtc5\" (UniqueName: \"kubernetes.io/projected/81021233-9462-48e8-9cb2-b4165344a604-kube-api-access-nrtc5\") pod \"cert-manager-webhook-6486ffd997-jr6b9\" (UID: \"81021233-9462-48e8-9cb2-b4165344a604\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr6b9" Jul 14 18:27:08.886067 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:08.885964 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtc5\" (UniqueName: \"kubernetes.io/projected/81021233-9462-48e8-9cb2-b4165344a604-kube-api-access-nrtc5\") pod \"cert-manager-webhook-6486ffd997-jr6b9\" (UID: \"81021233-9462-48e8-9cb2-b4165344a604\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr6b9" Jul 14 18:27:08.886067 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:08.886060 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81021233-9462-48e8-9cb2-b4165344a604-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jr6b9\" (UID: \"81021233-9462-48e8-9cb2-b4165344a604\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr6b9" Jul 14 18:27:08.905014 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:08.904983 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtc5\" (UniqueName: \"kubernetes.io/projected/81021233-9462-48e8-9cb2-b4165344a604-kube-api-access-nrtc5\") pod \"cert-manager-webhook-6486ffd997-jr6b9\" (UID: \"81021233-9462-48e8-9cb2-b4165344a604\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr6b9" Jul 14 18:27:08.906495 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:08.906472 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81021233-9462-48e8-9cb2-b4165344a604-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jr6b9\" (UID: \"81021233-9462-48e8-9cb2-b4165344a604\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jr6b9" Jul 14 18:27:08.927460 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:08.927430 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jr6b9" Jul 14 18:27:09.080481 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:09.080454 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jr6b9"] Jul 14 18:27:09.083254 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:27:09.083231 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81021233_9462_48e8_9cb2_b4165344a604.slice/crio-307669aeb4bf31b220f45848ccaddcb83dfd6984e15a2b10a7fe807d0eedd20c WatchSource:0}: Error finding container 307669aeb4bf31b220f45848ccaddcb83dfd6984e15a2b10a7fe807d0eedd20c: Status 404 returned error can't find the container with id 307669aeb4bf31b220f45848ccaddcb83dfd6984e15a2b10a7fe807d0eedd20c Jul 14 18:27:09.085230 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:09.085212 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:27:09.768055 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:09.768019 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jr6b9" event={"ID":"81021233-9462-48e8-9cb2-b4165344a604","Type":"ContainerStarted","Data":"307669aeb4bf31b220f45848ccaddcb83dfd6984e15a2b10a7fe807d0eedd20c"} Jul 14 18:27:10.528082 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:10.528021 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:10.728378 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:10.728280 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:12.712529 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:12.712501 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:12.784885 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:12.784854 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jr6b9" event={"ID":"81021233-9462-48e8-9cb2-b4165344a604","Type":"ContainerStarted","Data":"b5cbaf87bb46d0f8e3e062fb2b1755859543789994034fdadd96a22523dd129b"} Jul 14 18:27:12.785055 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:12.784916 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jr6b9" Jul 14 18:27:12.823413 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:12.823370 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-jr6b9" podStartSLOduration=1.681501746 podStartE2EDuration="4.823355116s" totalImagesPullingTime="3.14185337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:27:12.819278372 +0000 UTC m=+334.356726156" watchObservedRunningTime="2026-07-14 18:27:12.823355116 +0000 UTC m=+334.360802901" Jul 14 18:27:12.874298 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:12.874256 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:16.424049 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:16.424018 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z8m9t"] Jul 14 18:27:16.427579 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:16.427557 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8m9t" Jul 14 18:27:16.432715 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:16.432692 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-4hxmf\"" Jul 14 18:27:16.457207 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:16.457179 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z8m9t"] Jul 14 18:27:16.552743 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:16.552699 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d247\" (UniqueName: \"kubernetes.io/projected/12ddae8b-997f-489c-ae98-3feb9f6ab803-kube-api-access-4d247\") pod \"cert-manager-cainjector-559cdd8c87-z8m9t\" (UID: \"12ddae8b-997f-489c-ae98-3feb9f6ab803\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8m9t" Jul 14 18:27:16.552912 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:16.552780 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12ddae8b-997f-489c-ae98-3feb9f6ab803-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z8m9t\" (UID: \"12ddae8b-997f-489c-ae98-3feb9f6ab803\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8m9t" Jul 14 18:27:16.654156 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:16.654121 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d247\" (UniqueName: \"kubernetes.io/projected/12ddae8b-997f-489c-ae98-3feb9f6ab803-kube-api-access-4d247\") pod \"cert-manager-cainjector-559cdd8c87-z8m9t\" (UID: \"12ddae8b-997f-489c-ae98-3feb9f6ab803\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8m9t" Jul 14 18:27:16.654339 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:16.654183 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12ddae8b-997f-489c-ae98-3feb9f6ab803-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z8m9t\" (UID: \"12ddae8b-997f-489c-ae98-3feb9f6ab803\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8m9t" Jul 14 18:27:16.664313 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:16.664269 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12ddae8b-997f-489c-ae98-3feb9f6ab803-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z8m9t\" (UID: \"12ddae8b-997f-489c-ae98-3feb9f6ab803\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8m9t" Jul 14 18:27:16.664570 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:16.664549 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d247\" (UniqueName: \"kubernetes.io/projected/12ddae8b-997f-489c-ae98-3feb9f6ab803-kube-api-access-4d247\") pod \"cert-manager-cainjector-559cdd8c87-z8m9t\" (UID: \"12ddae8b-997f-489c-ae98-3feb9f6ab803\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8m9t" Jul 14 18:27:16.736940 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:16.736915 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8m9t" Jul 14 18:27:16.878383 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:16.878353 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z8m9t"] Jul 14 18:27:16.882060 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:27:16.882032 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12ddae8b_997f_489c_ae98_3feb9f6ab803.slice/crio-b5e19c3402f00d98e5cce7c2cb6d5b56849761274542022942528724b361d39e WatchSource:0}: Error finding container b5e19c3402f00d98e5cce7c2cb6d5b56849761274542022942528724b361d39e: Status 404 returned error can't find the container with id b5e19c3402f00d98e5cce7c2cb6d5b56849761274542022942528724b361d39e Jul 14 18:27:17.691822 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:17.691783 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:17.787844 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:17.787811 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:17.809159 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:17.809127 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8m9t" event={"ID":"12ddae8b-997f-489c-ae98-3feb9f6ab803","Type":"ContainerStarted","Data":"345bda827cb08d9324042cf6f8a36aacf4d885a062faa9e9e3cb3656e4a5b05f"} Jul 14 18:27:17.809301 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:17.809175 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8m9t" event={"ID":"12ddae8b-997f-489c-ae98-3feb9f6ab803","Type":"ContainerStarted","Data":"b5e19c3402f00d98e5cce7c2cb6d5b56849761274542022942528724b361d39e"} Jul 14 18:27:17.828829 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:17.828781 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z8m9t" podStartSLOduration=1.828766261 podStartE2EDuration="1.828766261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:27:17.82812374 +0000 UTC m=+339.365571526" watchObservedRunningTime="2026-07-14 18:27:17.828766261 +0000 UTC m=+339.366214045" Jul 14 18:27:18.096250 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:18.096167 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:18.791163 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:18.791064 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jr6b9" Jul 14 18:27:19.442765 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:19.442730 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qlrpt"] Jul 14 18:27:19.446087 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:19.446070 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qlrpt" Jul 14 18:27:19.449544 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:19.449526 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-n26mn\"" Jul 14 18:27:19.466804 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:19.466772 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qlrpt"] Jul 14 18:27:19.476597 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:19.476573 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh5h4\" (UniqueName: \"kubernetes.io/projected/aabbd72b-cdbd-4cf3-b4cb-bc06939e6fe1-kube-api-access-zh5h4\") pod \"cert-manager-55b4d954d5-qlrpt\" (UID: \"aabbd72b-cdbd-4cf3-b4cb-bc06939e6fe1\") " pod="cert-manager/cert-manager-55b4d954d5-qlrpt" Jul 14 18:27:19.476703 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:19.476620 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aabbd72b-cdbd-4cf3-b4cb-bc06939e6fe1-bound-sa-token\") pod \"cert-manager-55b4d954d5-qlrpt\" (UID: \"aabbd72b-cdbd-4cf3-b4cb-bc06939e6fe1\") " pod="cert-manager/cert-manager-55b4d954d5-qlrpt" Jul 14 18:27:19.577321 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:19.577272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh5h4\" (UniqueName: \"kubernetes.io/projected/aabbd72b-cdbd-4cf3-b4cb-bc06939e6fe1-kube-api-access-zh5h4\") pod \"cert-manager-55b4d954d5-qlrpt\" (UID: \"aabbd72b-cdbd-4cf3-b4cb-bc06939e6fe1\") " pod="cert-manager/cert-manager-55b4d954d5-qlrpt" Jul 14 18:27:19.577453 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:19.577372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aabbd72b-cdbd-4cf3-b4cb-bc06939e6fe1-bound-sa-token\") pod \"cert-manager-55b4d954d5-qlrpt\" (UID: \"aabbd72b-cdbd-4cf3-b4cb-bc06939e6fe1\") " pod="cert-manager/cert-manager-55b4d954d5-qlrpt" Jul 14 18:27:19.587301 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:19.587254 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh5h4\" (UniqueName: \"kubernetes.io/projected/aabbd72b-cdbd-4cf3-b4cb-bc06939e6fe1-kube-api-access-zh5h4\") pod \"cert-manager-55b4d954d5-qlrpt\" (UID: \"aabbd72b-cdbd-4cf3-b4cb-bc06939e6fe1\") " pod="cert-manager/cert-manager-55b4d954d5-qlrpt" Jul 14 18:27:19.589600 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:19.589576 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aabbd72b-cdbd-4cf3-b4cb-bc06939e6fe1-bound-sa-token\") pod \"cert-manager-55b4d954d5-qlrpt\" (UID: \"aabbd72b-cdbd-4cf3-b4cb-bc06939e6fe1\") " pod="cert-manager/cert-manager-55b4d954d5-qlrpt" Jul 14 18:27:19.755609 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:19.755542 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qlrpt" Jul 14 18:27:19.831868 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:19.831838 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:19.894182 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:19.894156 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qlrpt"] Jul 14 18:27:19.895934 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:27:19.895906 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaabbd72b_cdbd_4cf3_b4cb_bc06939e6fe1.slice/crio-f70b83854e953385894f36a9a76810be7366464f7438bf6981a4367dc05f6bdc WatchSource:0}: Error finding container f70b83854e953385894f36a9a76810be7366464f7438bf6981a4367dc05f6bdc: Status 404 returned error can't find the container with id f70b83854e953385894f36a9a76810be7366464f7438bf6981a4367dc05f6bdc Jul 14 18:27:20.825793 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:20.825751 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qlrpt" event={"ID":"aabbd72b-cdbd-4cf3-b4cb-bc06939e6fe1","Type":"ContainerStarted","Data":"286aac877b9eac225f2dca9fcd3bb8d3f66a6b5ff1059c3f2e7dcb08f26e0717"} Jul 14 18:27:20.825793 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:20.825800 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qlrpt" event={"ID":"aabbd72b-cdbd-4cf3-b4cb-bc06939e6fe1","Type":"ContainerStarted","Data":"f70b83854e953385894f36a9a76810be7366464f7438bf6981a4367dc05f6bdc"} Jul 14 18:27:20.850785 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:20.850745 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-qlrpt" podStartSLOduration=1.850733116 podStartE2EDuration="1.850733116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:27:20.847989791 +0000 UTC m=+342.385437575" watchObservedRunningTime="2026-07-14 18:27:20.850733116 +0000 UTC m=+342.388180901" Jul 14 18:27:21.362871 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:21.362842 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:24.593588 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:24.593554 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:24.747926 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:24.747892 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:26.429869 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:26.429836 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:39.016481 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:39.016399 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:39.426096 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:39.426018 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:39.712073 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:39.712041 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:40.256297 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.256262 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:40.624125 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.624046 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql"] Jul 14 18:27:40.627695 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.627678 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.642669 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.642651 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h5s8h\"" Jul 14 18:27:40.659239 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.659216 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 18:27:40.659993 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.659977 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 18:27:40.660177 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.660161 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 18:27:40.661424 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.661409 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 18:27:40.684067 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.684044 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql"] Jul 14 18:27:40.737653 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.737622 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:40.747516 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.747491 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/45be84e1-0678-4cb0-aaf1-8898001e3eeb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-5vfql\" (UID: \"45be84e1-0678-4cb0-aaf1-8898001e3eeb\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.747629 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.747526 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45be84e1-0678-4cb0-aaf1-8898001e3eeb-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-5vfql\" (UID: \"45be84e1-0678-4cb0-aaf1-8898001e3eeb\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.747629 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.747549 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/45be84e1-0678-4cb0-aaf1-8898001e3eeb-manifests\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-5vfql\" (UID: \"45be84e1-0678-4cb0-aaf1-8898001e3eeb\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.747713 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.747674 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45be84e1-0678-4cb0-aaf1-8898001e3eeb-tmp\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-5vfql\" (UID: \"45be84e1-0678-4cb0-aaf1-8898001e3eeb\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.747754 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.747726 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl478\" (UniqueName: \"kubernetes.io/projected/45be84e1-0678-4cb0-aaf1-8898001e3eeb-kube-api-access-jl478\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-5vfql\" (UID: \"45be84e1-0678-4cb0-aaf1-8898001e3eeb\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.848564 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.848529 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl478\" (UniqueName: \"kubernetes.io/projected/45be84e1-0678-4cb0-aaf1-8898001e3eeb-kube-api-access-jl478\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-5vfql\" (UID: \"45be84e1-0678-4cb0-aaf1-8898001e3eeb\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.848765 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.848606 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/45be84e1-0678-4cb0-aaf1-8898001e3eeb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-5vfql\" (UID: \"45be84e1-0678-4cb0-aaf1-8898001e3eeb\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.848765 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.848640 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45be84e1-0678-4cb0-aaf1-8898001e3eeb-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-5vfql\" (UID: \"45be84e1-0678-4cb0-aaf1-8898001e3eeb\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.848765 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.848674 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/45be84e1-0678-4cb0-aaf1-8898001e3eeb-manifests\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-5vfql\" (UID: \"45be84e1-0678-4cb0-aaf1-8898001e3eeb\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.848765 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.848727 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45be84e1-0678-4cb0-aaf1-8898001e3eeb-tmp\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-5vfql\" (UID: \"45be84e1-0678-4cb0-aaf1-8898001e3eeb\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.849231 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.849198 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/45be84e1-0678-4cb0-aaf1-8898001e3eeb-manifests\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-5vfql\" (UID: \"45be84e1-0678-4cb0-aaf1-8898001e3eeb\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.851175 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.851147 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45be84e1-0678-4cb0-aaf1-8898001e3eeb-tmp\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-5vfql\" (UID: \"45be84e1-0678-4cb0-aaf1-8898001e3eeb\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.851508 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.851491 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/45be84e1-0678-4cb0-aaf1-8898001e3eeb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-5vfql\" (UID: \"45be84e1-0678-4cb0-aaf1-8898001e3eeb\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.851591 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.851574 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/45be84e1-0678-4cb0-aaf1-8898001e3eeb-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-5vfql\" (UID: \"45be84e1-0678-4cb0-aaf1-8898001e3eeb\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.869545 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.869515 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl478\" (UniqueName: \"kubernetes.io/projected/45be84e1-0678-4cb0-aaf1-8898001e3eeb-kube-api-access-jl478\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-5vfql\" (UID: \"45be84e1-0678-4cb0-aaf1-8898001e3eeb\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:40.938036 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:40.937962 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:41.170382 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:41.170355 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql"] Jul 14 18:27:41.172647 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:27:41.172618 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45be84e1_0678_4cb0_aaf1_8898001e3eeb.slice/crio-f34760c39304207dd8d31bfd3bdddd61df43baad2fe228dbe9dc459d632da13b WatchSource:0}: Error finding container f34760c39304207dd8d31bfd3bdddd61df43baad2fe228dbe9dc459d632da13b: Status 404 returned error can't find the container with id f34760c39304207dd8d31bfd3bdddd61df43baad2fe228dbe9dc459d632da13b Jul 14 18:27:41.202212 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:41.202178 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:41.382832 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:41.382800 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:41.663169 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:41.663064 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:41.879271 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:41.879241 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:41.923526 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:41.923446 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" event={"ID":"45be84e1-0678-4cb0-aaf1-8898001e3eeb","Type":"ContainerStarted","Data":"f34760c39304207dd8d31bfd3bdddd61df43baad2fe228dbe9dc459d632da13b"} Jul 14 18:27:44.944430 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:44.944393 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" event={"ID":"45be84e1-0678-4cb0-aaf1-8898001e3eeb","Type":"ContainerStarted","Data":"f29f0275288a705f46fa0996fb87c7a927e7f91d68b5456b51c4f082a714a322"} Jul 14 18:27:45.951435 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:45.951397 2573 generic.go:356] "Generic (PLEG): container finished" podID="45be84e1-0678-4cb0-aaf1-8898001e3eeb" containerID="f29f0275288a705f46fa0996fb87c7a927e7f91d68b5456b51c4f082a714a322" exitCode=0 Jul 14 18:27:45.951808 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:45.951480 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" event={"ID":"45be84e1-0678-4cb0-aaf1-8898001e3eeb","Type":"ContainerDied","Data":"f29f0275288a705f46fa0996fb87c7a927e7f91d68b5456b51c4f082a714a322"} Jul 14 18:27:46.958732 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:46.958690 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" event={"ID":"45be84e1-0678-4cb0-aaf1-8898001e3eeb","Type":"ContainerStarted","Data":"c7b1ddc7b6ebd170e9a94175d6c45b24261e2abcffd7d262d0d63c83a56d7487"} Jul 14 18:27:46.959204 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:46.958836 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:27:47.093059 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:47.093002 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" podStartSLOduration=2.20939987 podStartE2EDuration="7.092985677s" totalImagesPullingTime="3.804347928s" totalInitContainerRuntime="1.079237879s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:27:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:27:47.091344515 +0000 UTC m=+368.628792301" watchObservedRunningTime="2026-07-14 18:27:47.092985677 +0000 UTC m=+368.630433522" Jul 14 18:27:51.923554 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:51.923521 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:53.373918 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:53.373878 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:53.632192 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:53.632086 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:27:57.965842 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:27:57.965806 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-5vfql" Jul 14 18:28:03.119801 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:03.119765 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:03.612158 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:03.612120 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:04.817347 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:04.817314 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fgbsf"] Jul 14 18:28:04.822401 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:04.822378 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fgbsf" Jul 14 18:28:04.827661 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:04.827639 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 14 18:28:04.832757 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:04.832734 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-tv5hx\"" Jul 14 18:28:04.870029 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:04.869996 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fgbsf"] Jul 14 18:28:04.945048 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:04.945009 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:04.970923 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:04.970890 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/460ba81a-42b2-48e0-9fb0-ac035c67fd36-cert\") pod \"odh-model-controller-7998c67dc4-fgbsf\" (UID: \"460ba81a-42b2-48e0-9fb0-ac035c67fd36\") " pod="opendatahub/odh-model-controller-7998c67dc4-fgbsf" Jul 14 18:28:04.971090 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:04.970970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrn6c\" (UniqueName: \"kubernetes.io/projected/460ba81a-42b2-48e0-9fb0-ac035c67fd36-kube-api-access-wrn6c\") pod \"odh-model-controller-7998c67dc4-fgbsf\" (UID: \"460ba81a-42b2-48e0-9fb0-ac035c67fd36\") " pod="opendatahub/odh-model-controller-7998c67dc4-fgbsf" Jul 14 18:28:05.072892 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:05.071980 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/460ba81a-42b2-48e0-9fb0-ac035c67fd36-cert\") pod \"odh-model-controller-7998c67dc4-fgbsf\" (UID: \"460ba81a-42b2-48e0-9fb0-ac035c67fd36\") " pod="opendatahub/odh-model-controller-7998c67dc4-fgbsf" Jul 14 18:28:05.072892 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:05.072067 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrn6c\" (UniqueName: \"kubernetes.io/projected/460ba81a-42b2-48e0-9fb0-ac035c67fd36-kube-api-access-wrn6c\") pod \"odh-model-controller-7998c67dc4-fgbsf\" (UID: \"460ba81a-42b2-48e0-9fb0-ac035c67fd36\") " pod="opendatahub/odh-model-controller-7998c67dc4-fgbsf" Jul 14 18:28:05.075807 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:05.075757 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/460ba81a-42b2-48e0-9fb0-ac035c67fd36-cert\") pod \"odh-model-controller-7998c67dc4-fgbsf\" (UID: \"460ba81a-42b2-48e0-9fb0-ac035c67fd36\") " pod="opendatahub/odh-model-controller-7998c67dc4-fgbsf" Jul 14 18:28:05.105347 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:05.105317 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrn6c\" (UniqueName: \"kubernetes.io/projected/460ba81a-42b2-48e0-9fb0-ac035c67fd36-kube-api-access-wrn6c\") pod \"odh-model-controller-7998c67dc4-fgbsf\" (UID: \"460ba81a-42b2-48e0-9fb0-ac035c67fd36\") " pod="opendatahub/odh-model-controller-7998c67dc4-fgbsf" Jul 14 18:28:05.134221 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:05.134177 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-fgbsf" Jul 14 18:28:05.241085 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:05.240968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:05.342339 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:05.342140 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-fgbsf"] Jul 14 18:28:05.345757 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:28:05.345722 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460ba81a_42b2_48e0_9fb0_ac035c67fd36.slice/crio-c2efa6be572bd59ddcc350488a1f9f0b6b6beee8fae6c7e41d7a8d244e26d228 WatchSource:0}: Error finding container c2efa6be572bd59ddcc350488a1f9f0b6b6beee8fae6c7e41d7a8d244e26d228: Status 404 returned error can't find the container with id c2efa6be572bd59ddcc350488a1f9f0b6b6beee8fae6c7e41d7a8d244e26d228 Jul 14 18:28:05.552345 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:05.552309 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:06.056015 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.055973 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fgbsf" event={"ID":"460ba81a-42b2-48e0-9fb0-ac035c67fd36","Type":"ContainerStarted","Data":"c2efa6be572bd59ddcc350488a1f9f0b6b6beee8fae6c7e41d7a8d244e26d228"} Jul 14 18:28:06.060005 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.059971 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hxhf5"] Jul 14 18:28:06.065256 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.065235 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-hxhf5" Jul 14 18:28:06.069612 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.069591 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-r6f2s\"" Jul 14 18:28:06.069706 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.069684 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 14 18:28:06.096668 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.096640 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hxhf5"] Jul 14 18:28:06.113444 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.113391 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:06.181416 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.181346 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7jjh\" (UniqueName: \"kubernetes.io/projected/b9077472-e6b7-4938-a5c1-6acf578d0e13-kube-api-access-x7jjh\") pod \"model-serving-api-54d89bd5fc-hxhf5\" (UID: \"b9077472-e6b7-4938-a5c1-6acf578d0e13\") " pod="opendatahub/model-serving-api-54d89bd5fc-hxhf5" Jul 14 18:28:06.181416 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.181410 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9077472-e6b7-4938-a5c1-6acf578d0e13-tls-certs\") pod \"model-serving-api-54d89bd5fc-hxhf5\" (UID: \"b9077472-e6b7-4938-a5c1-6acf578d0e13\") " pod="opendatahub/model-serving-api-54d89bd5fc-hxhf5" Jul 14 18:28:06.282408 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.282366 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7jjh\" (UniqueName: \"kubernetes.io/projected/b9077472-e6b7-4938-a5c1-6acf578d0e13-kube-api-access-x7jjh\") pod \"model-serving-api-54d89bd5fc-hxhf5\" (UID: \"b9077472-e6b7-4938-a5c1-6acf578d0e13\") " pod="opendatahub/model-serving-api-54d89bd5fc-hxhf5" Jul 14 18:28:06.282609 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.282440 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9077472-e6b7-4938-a5c1-6acf578d0e13-tls-certs\") pod \"model-serving-api-54d89bd5fc-hxhf5\" (UID: \"b9077472-e6b7-4938-a5c1-6acf578d0e13\") " pod="opendatahub/model-serving-api-54d89bd5fc-hxhf5" Jul 14 18:28:06.285486 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.285460 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9077472-e6b7-4938-a5c1-6acf578d0e13-tls-certs\") pod \"model-serving-api-54d89bd5fc-hxhf5\" (UID: \"b9077472-e6b7-4938-a5c1-6acf578d0e13\") " pod="opendatahub/model-serving-api-54d89bd5fc-hxhf5" Jul 14 18:28:06.306941 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.306868 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:06.308112 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.308084 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7jjh\" (UniqueName: \"kubernetes.io/projected/b9077472-e6b7-4938-a5c1-6acf578d0e13-kube-api-access-x7jjh\") pod \"model-serving-api-54d89bd5fc-hxhf5\" (UID: \"b9077472-e6b7-4938-a5c1-6acf578d0e13\") " pod="opendatahub/model-serving-api-54d89bd5fc-hxhf5" Jul 14 18:28:06.376365 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.376334 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-hxhf5" Jul 14 18:28:06.569741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:06.569713 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-hxhf5"] Jul 14 18:28:06.571775 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:28:06.571742 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9077472_e6b7_4938_a5c1_6acf578d0e13.slice/crio-2fe1c0cca870a7e7fefb2b31252c740d409c5af87f4e4f14c76759faa61ce55a WatchSource:0}: Error finding container 2fe1c0cca870a7e7fefb2b31252c740d409c5af87f4e4f14c76759faa61ce55a: Status 404 returned error can't find the container with id 2fe1c0cca870a7e7fefb2b31252c740d409c5af87f4e4f14c76759faa61ce55a Jul 14 18:28:07.062381 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:07.062341 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-hxhf5" event={"ID":"b9077472-e6b7-4938-a5c1-6acf578d0e13","Type":"ContainerStarted","Data":"2fe1c0cca870a7e7fefb2b31252c740d409c5af87f4e4f14c76759faa61ce55a"} Jul 14 18:28:07.530196 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:07.530132 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:07.717879 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:07.717847 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq"] Jul 14 18:28:07.727857 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:07.727835 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq" Jul 14 18:28:07.732409 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:07.732382 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mw7nl\"" Jul 14 18:28:07.733714 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:07.733688 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 14 18:28:07.733839 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:07.733738 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 14 18:28:07.751890 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:07.751864 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq"] Jul 14 18:28:07.898834 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:07.898743 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c-tmp\") pod \"kube-auth-proxy-5d44f677bf-zv7jq\" (UID: \"b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq" Jul 14 18:28:07.898996 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:07.898875 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c-tls-certs\") pod \"kube-auth-proxy-5d44f677bf-zv7jq\" (UID: \"b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq" Jul 14 18:28:07.898996 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:07.898964 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt2km\" (UniqueName: \"kubernetes.io/projected/b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c-kube-api-access-lt2km\") pod \"kube-auth-proxy-5d44f677bf-zv7jq\" (UID: \"b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq" Jul 14 18:28:08.000075 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:08.000042 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c-tmp\") pod \"kube-auth-proxy-5d44f677bf-zv7jq\" (UID: \"b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq" Jul 14 18:28:08.000248 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:08.000124 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c-tls-certs\") pod \"kube-auth-proxy-5d44f677bf-zv7jq\" (UID: \"b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq" Jul 14 18:28:08.000248 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:08.000193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt2km\" (UniqueName: \"kubernetes.io/projected/b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c-kube-api-access-lt2km\") pod \"kube-auth-proxy-5d44f677bf-zv7jq\" (UID: \"b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq" Jul 14 18:28:08.002379 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:08.002352 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c-tmp\") pod \"kube-auth-proxy-5d44f677bf-zv7jq\" (UID: \"b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq" Jul 14 18:28:08.002554 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:08.002536 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c-tls-certs\") pod \"kube-auth-proxy-5d44f677bf-zv7jq\" (UID: \"b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq" Jul 14 18:28:08.024174 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:08.024146 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt2km\" (UniqueName: \"kubernetes.io/projected/b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c-kube-api-access-lt2km\") pod \"kube-auth-proxy-5d44f677bf-zv7jq\" (UID: \"b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq" Jul 14 18:28:08.043029 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:08.042994 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq" Jul 14 18:28:08.682042 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:08.681989 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:09.173263 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:09.173226 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq"] Jul 14 18:28:09.178611 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:28:09.178585 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46b6c3a_b3d0_47c4_b198_3c9c1e8fcb1c.slice/crio-388b337aefce5ff32480d51f2b4e6abb1725e5ce2d8b393b13f57f92ea72572c WatchSource:0}: Error finding container 388b337aefce5ff32480d51f2b4e6abb1725e5ce2d8b393b13f57f92ea72572c: Status 404 returned error can't find the container with id 388b337aefce5ff32480d51f2b4e6abb1725e5ce2d8b393b13f57f92ea72572c Jul 14 18:28:10.080483 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.080442 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-hxhf5" event={"ID":"b9077472-e6b7-4938-a5c1-6acf578d0e13","Type":"ContainerStarted","Data":"b6a2588b4382904e609d730edc1e23efd092a6861ef77e83e402b804fb735edf"} Jul 14 18:28:10.080980 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.080572 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-hxhf5" Jul 14 18:28:10.082008 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.081983 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq" event={"ID":"b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c","Type":"ContainerStarted","Data":"388b337aefce5ff32480d51f2b4e6abb1725e5ce2d8b393b13f57f92ea72572c"} Jul 14 18:28:10.083644 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.083621 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-fgbsf" event={"ID":"460ba81a-42b2-48e0-9fb0-ac035c67fd36","Type":"ContainerStarted","Data":"5428b0404a72625a85a85ec3ad9a718cd7fb59d52329b9c67a15f7d2c217a855"} Jul 14 18:28:10.083751 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.083673 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-fgbsf" Jul 14 18:28:10.161487 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.161435 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-hxhf5" podStartSLOduration=1.736762766 podStartE2EDuration="4.161419613s" totalImagesPullingTime="2.424656847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:28:10.161008253 +0000 UTC m=+391.698456039" watchObservedRunningTime="2026-07-14 18:28:10.161419613 +0000 UTC m=+391.698867398" Jul 14 18:28:10.268432 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.268378 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-fgbsf" podStartSLOduration=2.613425473 podStartE2EDuration="6.268359264s" totalImagesPullingTime="3.654933791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:28:10.264123946 +0000 UTC m=+391.801571732" watchObservedRunningTime="2026-07-14 18:28:10.268359264 +0000 UTC m=+391.805807056" Jul 14 18:28:10.514567 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.514532 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n"] Jul 14 18:28:10.517267 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.517239 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:10.518493 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.518472 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.531319 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.531276 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 14 18:28:10.531550 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.531491 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 18:28:10.531550 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.531525 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 14 18:28:10.531550 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.531546 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 14 18:28:10.531868 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.531659 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jgq6l\"" Jul 14 18:28:10.629066 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.629029 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-cacerts\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.629066 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.629065 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.629253 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.629101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae6b296-e53a-4e16-99ad-faff178e180b-istio-token\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.629253 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.629117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.629253 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.629146 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae6b296-e53a-4e16-99ad-faff178e180b-local-certs\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.629253 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.629162 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gspr9\" (UniqueName: \"kubernetes.io/projected/cae6b296-e53a-4e16-99ad-faff178e180b-kube-api-access-gspr9\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.629253 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.629234 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cae6b296-e53a-4e16-99ad-faff178e180b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.694750 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.694713 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n"] Jul 14 18:28:10.702224 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.702185 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:10.730224 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.730188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae6b296-e53a-4e16-99ad-faff178e180b-istio-token\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.730224 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.730227 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.730446 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.730259 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae6b296-e53a-4e16-99ad-faff178e180b-local-certs\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.730515 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.730495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gspr9\" (UniqueName: \"kubernetes.io/projected/cae6b296-e53a-4e16-99ad-faff178e180b-kube-api-access-gspr9\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.730596 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.730571 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cae6b296-e53a-4e16-99ad-faff178e180b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.730721 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.730695 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-cacerts\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.730869 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.730729 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.731266 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.731243 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cae6b296-e53a-4e16-99ad-faff178e180b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.732940 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.732899 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.733317 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.733247 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae6b296-e53a-4e16-99ad-faff178e180b-local-certs\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.733435 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.733412 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-cacerts\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.733538 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.733460 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.750206 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.750184 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gspr9\" (UniqueName: \"kubernetes.io/projected/cae6b296-e53a-4e16-99ad-faff178e180b-kube-api-access-gspr9\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.750334 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.750249 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae6b296-e53a-4e16-99ad-faff178e180b-istio-token\") pod \"istiod-openshift-gateway-7574459656-xgf6n\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:10.831664 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:10.831566 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:11.073277 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:11.073249 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n"] Jul 14 18:28:11.076628 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:28:11.076601 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae6b296_e53a_4e16_99ad_faff178e180b.slice/crio-e6209a03452886c3136cfdbbb970e6c2a691b7ab6f4144254326a5f8f0362b90 WatchSource:0}: Error finding container e6209a03452886c3136cfdbbb970e6c2a691b7ab6f4144254326a5f8f0362b90: Status 404 returned error can't find the container with id e6209a03452886c3136cfdbbb970e6c2a691b7ab6f4144254326a5f8f0362b90 Jul 14 18:28:11.091379 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:11.091320 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" event={"ID":"cae6b296-e53a-4e16-99ad-faff178e180b","Type":"ContainerStarted","Data":"e6209a03452886c3136cfdbbb970e6c2a691b7ab6f4144254326a5f8f0362b90"} Jul 14 18:28:13.105369 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:13.105328 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq" event={"ID":"b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c","Type":"ContainerStarted","Data":"27c6239c6b1196159b62eeed96080b8498a550a163403a52f8f355c5ba2693a7"} Jul 14 18:28:13.225007 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:13.224968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:13.275666 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:13.275615 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d44f677bf-zv7jq" podStartSLOduration=3.106084544 podStartE2EDuration="6.275598723s" totalImagesPullingTime="3.169514179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:28:13.273213456 +0000 UTC m=+394.810661240" watchObservedRunningTime="2026-07-14 18:28:13.275598723 +0000 UTC m=+394.813046507" Jul 14 18:28:14.597319 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:14.597260 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:28:14.597603 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:14.597341 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:28:15.120924 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:15.120890 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" event={"ID":"cae6b296-e53a-4e16-99ad-faff178e180b","Type":"ContainerStarted","Data":"efa60b16ab4acf3bfce25fa31e5f16b520be1ae765fdab3051d3cbc66151ec35"} Jul 14 18:28:15.121109 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:15.120955 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:15.123005 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:15.122977 2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-xgf6n container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 18:28:15.123145 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:15.123027 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" podUID="cae6b296-e53a-4e16-99ad-faff178e180b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 18:28:15.331066 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:15.331009 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" podStartSLOduration=1.812788713 podStartE2EDuration="5.330992406s" totalImagesPullingTime="3.518203693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:28:15.328274069 +0000 UTC m=+396.865721856" watchObservedRunningTime="2026-07-14 18:28:15.330992406 +0000 UTC m=+396.868440190" Jul 14 18:28:15.541147 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:15.541112 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:15.809865 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:15.809791 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:16.127023 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:16.126947 2573 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-xgf6n container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 18:28:16.127023 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:16.127012 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" podUID="cae6b296-e53a-4e16-99ad-faff178e180b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 18:28:19.016607 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.016575 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj"] Jul 14 18:28:19.025923 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.025893 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.033215 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.033193 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hxzg7\"" Jul 14 18:28:19.101649 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.101622 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj"] Jul 14 18:28:19.116793 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.116768 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b9b6de-ac61-470e-948e-e48e88cf4039-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.116793 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.116796 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b9b6de-ac61-470e-948e-e48e88cf4039-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.116983 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.116845 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b9b6de-ac61-470e-948e-e48e88cf4039-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.116983 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.116880 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b9b6de-ac61-470e-948e-e48e88cf4039-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.116983 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.116921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b9b6de-ac61-470e-948e-e48e88cf4039-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.116983 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.116974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b9b6de-ac61-470e-948e-e48e88cf4039-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.117115 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.117008 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b9b6de-ac61-470e-948e-e48e88cf4039-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.117115 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.117038 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b9b6de-ac61-470e-948e-e48e88cf4039-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.117115 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.117061 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkjl9\" (UniqueName: \"kubernetes.io/projected/99b9b6de-ac61-470e-948e-e48e88cf4039-kube-api-access-jkjl9\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.127239 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.127212 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:19.181614 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.181564 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:19.217594 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.217554 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b9b6de-ac61-470e-948e-e48e88cf4039-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.217744 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.217635 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b9b6de-ac61-470e-948e-e48e88cf4039-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.217744 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.217699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b9b6de-ac61-470e-948e-e48e88cf4039-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.217880 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.217748 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b9b6de-ac61-470e-948e-e48e88cf4039-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.217880 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.217805 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b9b6de-ac61-470e-948e-e48e88cf4039-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.217880 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.217840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b9b6de-ac61-470e-948e-e48e88cf4039-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.217880 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.217875 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkjl9\" (UniqueName: \"kubernetes.io/projected/99b9b6de-ac61-470e-948e-e48e88cf4039-kube-api-access-jkjl9\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.218073 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.217940 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b9b6de-ac61-470e-948e-e48e88cf4039-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.218073 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.217960 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b9b6de-ac61-470e-948e-e48e88cf4039-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.218181 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.217965 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b9b6de-ac61-470e-948e-e48e88cf4039-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.218556 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.218523 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b9b6de-ac61-470e-948e-e48e88cf4039-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.218556 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.218551 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b9b6de-ac61-470e-948e-e48e88cf4039-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.218735 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.218635 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b9b6de-ac61-470e-948e-e48e88cf4039-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.219107 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.219066 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b9b6de-ac61-470e-948e-e48e88cf4039-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.220734 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.220709 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b9b6de-ac61-470e-948e-e48e88cf4039-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.221038 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.221005 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b9b6de-ac61-470e-948e-e48e88cf4039-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.272204 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.272143 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b9b6de-ac61-470e-948e-e48e88cf4039-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.272486 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.272463 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkjl9\" (UniqueName: \"kubernetes.io/projected/99b9b6de-ac61-470e-948e-e48e88cf4039-kube-api-access-jkjl9\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4tgcnj\" (UID: \"99b9b6de-ac61-470e-948e-e48e88cf4039\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.336561 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.336529 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:19.419429 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.419400 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:19.718163 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.718052 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj"] Jul 14 18:28:19.728017 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:28:19.727989 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b9b6de_ac61_470e_948e_e48e88cf4039.slice/crio-9b3bd86d624fe6b999813b7130ef509a8954c863ec0cd447b5bdb7628c10cfab WatchSource:0}: Error finding container 9b3bd86d624fe6b999813b7130ef509a8954c863ec0cd447b5bdb7628c10cfab: Status 404 returned error can't find the container with id 9b3bd86d624fe6b999813b7130ef509a8954c863ec0cd447b5bdb7628c10cfab Jul 14 18:28:19.835752 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:19.835717 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:20.147873 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:20.147782 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" event={"ID":"99b9b6de-ac61-470e-948e-e48e88cf4039","Type":"ContainerStarted","Data":"9b3bd86d624fe6b999813b7130ef509a8954c863ec0cd447b5bdb7628c10cfab"} Jul 14 18:28:21.094069 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:21.094038 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-fgbsf" Jul 14 18:28:21.097131 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:21.097108 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-hxhf5" Jul 14 18:28:22.588720 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:22.588672 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:28:22.589008 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:22.588774 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:28:22.589008 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:22.588820 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:28:23.173693 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:23.173659 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" event={"ID":"99b9b6de-ac61-470e-948e-e48e88cf4039","Type":"ContainerStarted","Data":"87f4ce29d14642966f06ccdcd36ba296ec47f64ebad51fe9cd27a04da1d18c60"} Jul 14 18:28:23.336846 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:23.336809 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:23.341440 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:23.341413 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:23.520338 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:23.520275 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:23.580512 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:23.580460 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" podStartSLOduration=2.722022547 podStartE2EDuration="5.580443587s" totalImagesPullingTime="2.85842104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:28:23.518807443 +0000 UTC m=+405.056255234" watchObservedRunningTime="2026-07-14 18:28:23.580443587 +0000 UTC m=+405.117891445" Jul 14 18:28:23.941147 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:23.941112 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n"] Jul 14 18:28:23.941535 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:23.941316 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" podUID="cae6b296-e53a-4e16-99ad-faff178e180b" containerName="discovery" containerID="cri-o://efa60b16ab4acf3bfce25fa31e5f16b520be1ae765fdab3051d3cbc66151ec35" gracePeriod=30 Jul 14 18:28:24.086232 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.086182 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:24.180195 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.180162 2573 generic.go:356] "Generic (PLEG): container finished" podID="cae6b296-e53a-4e16-99ad-faff178e180b" containerID="efa60b16ab4acf3bfce25fa31e5f16b520be1ae765fdab3051d3cbc66151ec35" exitCode=0 Jul 14 18:28:24.180345 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.180236 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" event={"ID":"cae6b296-e53a-4e16-99ad-faff178e180b","Type":"ContainerDied","Data":"efa60b16ab4acf3bfce25fa31e5f16b520be1ae765fdab3051d3cbc66151ec35"} Jul 14 18:28:24.180345 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.180305 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" event={"ID":"cae6b296-e53a-4e16-99ad-faff178e180b","Type":"ContainerDied","Data":"e6209a03452886c3136cfdbbb970e6c2a691b7ab6f4144254326a5f8f0362b90"} Jul 14 18:28:24.180345 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.180326 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6209a03452886c3136cfdbbb970e6c2a691b7ab6f4144254326a5f8f0362b90" Jul 14 18:28:24.180639 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.180625 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:24.181596 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.181577 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4tgcnj" Jul 14 18:28:24.186093 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.186074 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:24.261558 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.261481 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-46m4j"] Jul 14 18:28:24.261889 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.261877 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cae6b296-e53a-4e16-99ad-faff178e180b" containerName="discovery" Jul 14 18:28:24.261928 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.261890 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cae6b296-e53a-4e16-99ad-faff178e180b" containerName="discovery" Jul 14 18:28:24.261959 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.261954 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cae6b296-e53a-4e16-99ad-faff178e180b" containerName="discovery" Jul 14 18:28:24.264938 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.264912 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46m4j" Jul 14 18:28:24.270256 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.270229 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-cacerts\") pod \"cae6b296-e53a-4e16-99ad-faff178e180b\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " Jul 14 18:28:24.270409 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.270270 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae6b296-e53a-4e16-99ad-faff178e180b-local-certs\") pod \"cae6b296-e53a-4e16-99ad-faff178e180b\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " Jul 14 18:28:24.270409 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.270335 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-istio-csr-dns-cert\") pod \"cae6b296-e53a-4e16-99ad-faff178e180b\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " Jul 14 18:28:24.270409 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.270373 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cae6b296-e53a-4e16-99ad-faff178e180b-istio-csr-ca-configmap\") pod \"cae6b296-e53a-4e16-99ad-faff178e180b\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " Jul 14 18:28:24.270576 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.270440 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gspr9\" (UniqueName: \"kubernetes.io/projected/cae6b296-e53a-4e16-99ad-faff178e180b-kube-api-access-gspr9\") pod \"cae6b296-e53a-4e16-99ad-faff178e180b\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " Jul 14 18:28:24.270576 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.270479 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-istio-kubeconfig\") pod \"cae6b296-e53a-4e16-99ad-faff178e180b\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " Jul 14 18:28:24.270576 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.270511 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae6b296-e53a-4e16-99ad-faff178e180b-istio-token\") pod \"cae6b296-e53a-4e16-99ad-faff178e180b\" (UID: \"cae6b296-e53a-4e16-99ad-faff178e180b\") " Jul 14 18:28:24.271688 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.271025 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae6b296-e53a-4e16-99ad-faff178e180b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "cae6b296-e53a-4e16-99ad-faff178e180b" (UID: "cae6b296-e53a-4e16-99ad-faff178e180b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:28:24.271688 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.271207 2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cae6b296-e53a-4e16-99ad-faff178e180b-istio-csr-ca-configmap\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:28:24.273171 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.273144 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "cae6b296-e53a-4e16-99ad-faff178e180b" (UID: "cae6b296-e53a-4e16-99ad-faff178e180b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:28:24.273318 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.273233 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae6b296-e53a-4e16-99ad-faff178e180b-local-certs" (OuterVolumeSpecName: "local-certs") pod "cae6b296-e53a-4e16-99ad-faff178e180b" (UID: "cae6b296-e53a-4e16-99ad-faff178e180b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:28:24.274105 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.273853 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-cacerts" (OuterVolumeSpecName: "cacerts") pod "cae6b296-e53a-4e16-99ad-faff178e180b" (UID: "cae6b296-e53a-4e16-99ad-faff178e180b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:28:24.274105 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.273935 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae6b296-e53a-4e16-99ad-faff178e180b-kube-api-access-gspr9" (OuterVolumeSpecName: "kube-api-access-gspr9") pod "cae6b296-e53a-4e16-99ad-faff178e180b" (UID: "cae6b296-e53a-4e16-99ad-faff178e180b"). InnerVolumeSpecName "kube-api-access-gspr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:28:24.274105 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.274098 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae6b296-e53a-4e16-99ad-faff178e180b-istio-token" (OuterVolumeSpecName: "istio-token") pod "cae6b296-e53a-4e16-99ad-faff178e180b" (UID: "cae6b296-e53a-4e16-99ad-faff178e180b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:28:24.274304 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.274114 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "cae6b296-e53a-4e16-99ad-faff178e180b" (UID: "cae6b296-e53a-4e16-99ad-faff178e180b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:28:24.279362 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.279341 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zv8b9\"" Jul 14 18:28:24.279642 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.279625 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 14 18:28:24.358410 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.358365 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-46m4j"] Jul 14 18:28:24.360102 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.360085 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx"] Jul 14 18:28:24.363804 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.363790 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx" Jul 14 18:28:24.371968 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.371945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmjt\" (UniqueName: \"kubernetes.io/projected/8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d-kube-api-access-5vmjt\") pod \"kserve-controller-manager-d6f4b65b8-46m4j\" (UID: \"8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46m4j" Jul 14 18:28:24.372082 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.371976 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d-cert\") pod \"kserve-controller-manager-d6f4b65b8-46m4j\" (UID: \"8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46m4j" Jul 14 18:28:24.372082 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.372048 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gspr9\" (UniqueName: \"kubernetes.io/projected/cae6b296-e53a-4e16-99ad-faff178e180b-kube-api-access-gspr9\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:28:24.372082 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.372060 2573 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-istio-kubeconfig\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:28:24.372082 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.372070 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae6b296-e53a-4e16-99ad-faff178e180b-istio-token\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:28:24.372082 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.372080 2573 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-cacerts\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:28:24.372265 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.372094 2573 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae6b296-e53a-4e16-99ad-faff178e180b-local-certs\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:28:24.372265 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.372105 2573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cae6b296-e53a-4e16-99ad-faff178e180b-istio-csr-dns-cert\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:28:24.381216 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:28:24.381194 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"llmisvc-webhook-server-cert\" is forbidden: User \"system:node:ip-10-0-132-224.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-224.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" type="*v1.Secret" Jul 14 18:28:24.384208 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:28:24.384180 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"llmisvc-controller-manager-dockercfg-dskbx\" is forbidden: User \"system:node:ip-10-0-132-224.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-224.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dskbx\"" type="*v1.Secret" Jul 14 18:28:24.472877 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.472845 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmjt\" (UniqueName: \"kubernetes.io/projected/8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d-kube-api-access-5vmjt\") pod \"kserve-controller-manager-d6f4b65b8-46m4j\" (UID: \"8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46m4j" Jul 14 18:28:24.473037 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.472883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d-cert\") pod \"kserve-controller-manager-d6f4b65b8-46m4j\" (UID: \"8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46m4j" Jul 14 18:28:24.473037 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.472951 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcce408f-a62a-426c-afce-01250b0490ec-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-knkxx\" (UID: \"fcce408f-a62a-426c-afce-01250b0490ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx" Jul 14 18:28:24.473147 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.473032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s465q\" (UniqueName: \"kubernetes.io/projected/fcce408f-a62a-426c-afce-01250b0490ec-kube-api-access-s465q\") pod \"llmisvc-controller-manager-777c6bd7d5-knkxx\" (UID: \"fcce408f-a62a-426c-afce-01250b0490ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx" Jul 14 18:28:24.475466 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.475443 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d-cert\") pod \"kserve-controller-manager-d6f4b65b8-46m4j\" (UID: \"8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46m4j" Jul 14 18:28:24.533150 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.533086 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmjt\" (UniqueName: \"kubernetes.io/projected/8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d-kube-api-access-5vmjt\") pod \"kserve-controller-manager-d6f4b65b8-46m4j\" (UID: \"8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46m4j" Jul 14 18:28:24.573823 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.573800 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcce408f-a62a-426c-afce-01250b0490ec-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-knkxx\" (UID: \"fcce408f-a62a-426c-afce-01250b0490ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx" Jul 14 18:28:24.573952 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.573890 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s465q\" (UniqueName: \"kubernetes.io/projected/fcce408f-a62a-426c-afce-01250b0490ec-kube-api-access-s465q\") pod \"llmisvc-controller-manager-777c6bd7d5-knkxx\" (UID: \"fcce408f-a62a-426c-afce-01250b0490ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx" Jul 14 18:28:24.581808 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.581777 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx"] Jul 14 18:28:24.587493 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.587472 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46m4j" Jul 14 18:28:24.627632 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.627604 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s465q\" (UniqueName: \"kubernetes.io/projected/fcce408f-a62a-426c-afce-01250b0490ec-kube-api-access-s465q\") pod \"llmisvc-controller-manager-777c6bd7d5-knkxx\" (UID: \"fcce408f-a62a-426c-afce-01250b0490ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx" Jul 14 18:28:24.709450 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:24.709408 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:25.052242 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:25.052208 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-46m4j"] Jul 14 18:28:25.055966 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:28:25.055926 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bdf5bf9_9f6b_40b7_8afd_1c6dd47dab0d.slice/crio-59cb7b08117bc3ee301396163896723b4926cc8e43a4ca38fe2cc2c1038ff9cc WatchSource:0}: Error finding container 59cb7b08117bc3ee301396163896723b4926cc8e43a4ca38fe2cc2c1038ff9cc: Status 404 returned error can't find the container with id 59cb7b08117bc3ee301396163896723b4926cc8e43a4ca38fe2cc2c1038ff9cc Jul 14 18:28:25.185996 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:25.185956 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46m4j" event={"ID":"8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d","Type":"ContainerStarted","Data":"59cb7b08117bc3ee301396163896723b4926cc8e43a4ca38fe2cc2c1038ff9cc"} Jul 14 18:28:25.186173 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:25.186047 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n" Jul 14 18:28:25.230198 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:25.230172 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n"] Jul 14 18:28:25.232094 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:25.232074 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xgf6n"] Jul 14 18:28:25.410187 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:25.410105 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 14 18:28:25.416392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:25.416368 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fcce408f-a62a-426c-afce-01250b0490ec-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-knkxx\" (UID: \"fcce408f-a62a-426c-afce-01250b0490ec\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx" Jul 14 18:28:25.729756 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:25.729726 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-dskbx\"" Jul 14 18:28:25.733355 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:25.733331 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx" Jul 14 18:28:26.084947 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:26.084672 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:26.087475 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:26.087446 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx"] Jul 14 18:28:26.088307 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:28:26.088253 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfcce408f_a62a_426c_afce_01250b0490ec.slice/crio-ee69209677e24a2c6c269b665313a648e2aed1f3788cbb9ce34af5a29d1c182a WatchSource:0}: Error finding container ee69209677e24a2c6c269b665313a648e2aed1f3788cbb9ce34af5a29d1c182a: Status 404 returned error can't find the container with id ee69209677e24a2c6c269b665313a648e2aed1f3788cbb9ce34af5a29d1c182a Jul 14 18:28:26.198782 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:26.198733 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx" event={"ID":"fcce408f-a62a-426c-afce-01250b0490ec","Type":"ContainerStarted","Data":"ee69209677e24a2c6c269b665313a648e2aed1f3788cbb9ce34af5a29d1c182a"} Jul 14 18:28:26.649729 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:26.649691 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:28:27.222830 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:27.222796 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cae6b296-e53a-4e16-99ad-faff178e180b" path="/var/lib/kubelet/pods/cae6b296-e53a-4e16-99ad-faff178e180b/volumes" Jul 14 18:28:28.212664 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:28.212568 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46m4j" event={"ID":"8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d","Type":"ContainerStarted","Data":"c7771d6725b1cca0a0454b35ce227ea83c9ecb90dc1c84fde7c620873a9c3991"} Jul 14 18:28:28.212830 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:28.212659 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46m4j" Jul 14 18:28:28.265796 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:28.265749 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46m4j" podStartSLOduration=1.256044873 podStartE2EDuration="4.265734669s" totalImagesPullingTime="3.009689796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:28:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:28:28.262779595 +0000 UTC m=+409.800227381" watchObservedRunningTime="2026-07-14 18:28:28.265734669 +0000 UTC m=+409.803182454" Jul 14 18:28:30.225752 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:30.225717 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx" event={"ID":"fcce408f-a62a-426c-afce-01250b0490ec","Type":"ContainerStarted","Data":"780d1284d78d453fd86e09023efe3b4ce392e3bcb964f3c75f71f506030bf514"} Jul 14 18:28:30.226237 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:30.225834 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx" Jul 14 18:28:30.303745 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:30.303693 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx" podStartSLOduration=2.658638236 podStartE2EDuration="6.303675871s" totalImagesPullingTime="3.645037635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:28:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:28:30.299161767 +0000 UTC m=+411.836609552" watchObservedRunningTime="2026-07-14 18:28:30.303675871 +0000 UTC m=+411.841123655" Jul 14 18:28:59.225594 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:28:59.225567 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46m4j" Jul 14 18:29:01.233744 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:01.233669 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-knkxx" Jul 14 18:29:05.017682 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.017641 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:05.135180 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.135145 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8svfr"] Jul 14 18:29:05.140487 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.140463 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8svfr" Jul 14 18:29:05.143586 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.143559 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 18:29:05.144963 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.144944 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 18:29:05.147419 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.147398 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-l6drd\"" Jul 14 18:29:05.166826 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.166803 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8svfr"] Jul 14 18:29:05.178599 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.178567 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:05.228920 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.228897 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjlnt\" (UniqueName: \"kubernetes.io/projected/fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4-kube-api-access-bjlnt\") pod \"kuadrant-operator-catalog-8svfr\" (UID: \"fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4\") " pod="kuadrant-system/kuadrant-operator-catalog-8svfr" Jul 14 18:29:05.330187 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.330118 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjlnt\" (UniqueName: \"kubernetes.io/projected/fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4-kube-api-access-bjlnt\") pod \"kuadrant-operator-catalog-8svfr\" (UID: \"fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4\") " pod="kuadrant-system/kuadrant-operator-catalog-8svfr" Jul 14 18:29:05.357981 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.357958 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjlnt\" (UniqueName: \"kubernetes.io/projected/fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4-kube-api-access-bjlnt\") pod \"kuadrant-operator-catalog-8svfr\" (UID: \"fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4\") " pod="kuadrant-system/kuadrant-operator-catalog-8svfr" Jul 14 18:29:05.438532 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.438493 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8svfr"] Jul 14 18:29:05.438823 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.438807 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8svfr" Jul 14 18:29:05.514136 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.514109 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:05.634131 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.634109 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8svfr"] Jul 14 18:29:05.636619 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:29:05.636594 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac2c5bf_e37c_49ba_8e61_e1eaaf5e18b4.slice/crio-3e7a0d4de141dcfd760b0d9b5eda110579ceb05bdd0de84e222273d9a238ab8d WatchSource:0}: Error finding container 3e7a0d4de141dcfd760b0d9b5eda110579ceb05bdd0de84e222273d9a238ab8d: Status 404 returned error can't find the container with id 3e7a0d4de141dcfd760b0d9b5eda110579ceb05bdd0de84e222273d9a238ab8d Jul 14 18:29:05.668788 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.668753 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jhn8w"] Jul 14 18:29:05.673536 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.673515 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jhn8w" Jul 14 18:29:05.690370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.690343 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jhn8w"] Jul 14 18:29:05.733585 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.733554 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lj89\" (UniqueName: \"kubernetes.io/projected/7a5e68e3-6771-4f17-bb6b-88d2d5f0460f-kube-api-access-4lj89\") pod \"kuadrant-operator-catalog-jhn8w\" (UID: \"7a5e68e3-6771-4f17-bb6b-88d2d5f0460f\") " pod="kuadrant-system/kuadrant-operator-catalog-jhn8w" Jul 14 18:29:05.835123 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.835093 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lj89\" (UniqueName: \"kubernetes.io/projected/7a5e68e3-6771-4f17-bb6b-88d2d5f0460f-kube-api-access-4lj89\") pod \"kuadrant-operator-catalog-jhn8w\" (UID: \"7a5e68e3-6771-4f17-bb6b-88d2d5f0460f\") " pod="kuadrant-system/kuadrant-operator-catalog-jhn8w" Jul 14 18:29:05.853079 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.853042 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lj89\" (UniqueName: \"kubernetes.io/projected/7a5e68e3-6771-4f17-bb6b-88d2d5f0460f-kube-api-access-4lj89\") pod \"kuadrant-operator-catalog-jhn8w\" (UID: \"7a5e68e3-6771-4f17-bb6b-88d2d5f0460f\") " pod="kuadrant-system/kuadrant-operator-catalog-jhn8w" Jul 14 18:29:05.982694 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:05.982661 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jhn8w" Jul 14 18:29:06.138871 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:06.138850 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jhn8w"] Jul 14 18:29:06.140590 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:29:06.140568 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5e68e3_6771_4f17_bb6b_88d2d5f0460f.slice/crio-d60dd77f278cb30978d603c2c5a84ba2769207db9c4f48dbeb6457170d35a298 WatchSource:0}: Error finding container d60dd77f278cb30978d603c2c5a84ba2769207db9c4f48dbeb6457170d35a298: Status 404 returned error can't find the container with id d60dd77f278cb30978d603c2c5a84ba2769207db9c4f48dbeb6457170d35a298 Jul 14 18:29:06.408990 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:06.408911 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jhn8w" event={"ID":"7a5e68e3-6771-4f17-bb6b-88d2d5f0460f","Type":"ContainerStarted","Data":"d60dd77f278cb30978d603c2c5a84ba2769207db9c4f48dbeb6457170d35a298"} Jul 14 18:29:06.410313 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:06.410252 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8svfr" event={"ID":"fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4","Type":"ContainerStarted","Data":"3e7a0d4de141dcfd760b0d9b5eda110579ceb05bdd0de84e222273d9a238ab8d"} Jul 14 18:29:08.425517 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:08.425477 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jhn8w" event={"ID":"7a5e68e3-6771-4f17-bb6b-88d2d5f0460f","Type":"ContainerStarted","Data":"6342d8c18d9628c51296430eba7241bc45fcbe80cf3adffeec286d84b7861498"} Jul 14 18:29:08.426916 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:08.426894 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8svfr" event={"ID":"fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4","Type":"ContainerStarted","Data":"8bb69c2d9a359c701412949e83f446f7e8c8934bae695b2dfca8725dc709dd2d"} Jul 14 18:29:08.427024 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:08.426986 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-8svfr" podUID="fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4" containerName="registry-server" containerID="cri-o://8bb69c2d9a359c701412949e83f446f7e8c8934bae695b2dfca8725dc709dd2d" gracePeriod=2 Jul 14 18:29:08.504365 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:08.504319 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-jhn8w" podStartSLOduration=1.7222126850000001 podStartE2EDuration="3.504276217s" totalImagesPullingTime="1.782063532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:29:08.502453653 +0000 UTC m=+450.039901437" watchObservedRunningTime="2026-07-14 18:29:08.504276217 +0000 UTC m=+450.041724029" Jul 14 18:29:08.548867 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:08.548826 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8svfr" podStartSLOduration=1.265598858 podStartE2EDuration="3.548810466s" totalImagesPullingTime="2.283211608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:29:08.546970634 +0000 UTC m=+450.084418422" watchObservedRunningTime="2026-07-14 18:29:08.548810466 +0000 UTC m=+450.086258256" Jul 14 18:29:08.691204 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:08.691182 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8svfr" Jul 14 18:29:08.765440 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:08.765413 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjlnt\" (UniqueName: \"kubernetes.io/projected/fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4-kube-api-access-bjlnt\") pod \"fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4\" (UID: \"fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4\") " Jul 14 18:29:08.767550 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:08.767526 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4-kube-api-access-bjlnt" (OuterVolumeSpecName: "kube-api-access-bjlnt") pod "fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4" (UID: "fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4"). InnerVolumeSpecName "kube-api-access-bjlnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:29:08.866527 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:08.866497 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjlnt\" (UniqueName: \"kubernetes.io/projected/fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4-kube-api-access-bjlnt\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:29:09.433633 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:09.433599 2573 generic.go:356] "Generic (PLEG): container finished" podID="fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4" containerID="8bb69c2d9a359c701412949e83f446f7e8c8934bae695b2dfca8725dc709dd2d" exitCode=0 Jul 14 18:29:09.434025 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:09.433654 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8svfr" Jul 14 18:29:09.434025 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:09.433692 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8svfr" event={"ID":"fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4","Type":"ContainerDied","Data":"8bb69c2d9a359c701412949e83f446f7e8c8934bae695b2dfca8725dc709dd2d"} Jul 14 18:29:09.434025 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:09.433737 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8svfr" event={"ID":"fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4","Type":"ContainerDied","Data":"3e7a0d4de141dcfd760b0d9b5eda110579ceb05bdd0de84e222273d9a238ab8d"} Jul 14 18:29:09.434025 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:09.433778 2573 scope.go:117] "RemoveContainer" containerID="8bb69c2d9a359c701412949e83f446f7e8c8934bae695b2dfca8725dc709dd2d" Jul 14 18:29:09.443349 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:09.443326 2573 scope.go:117] "RemoveContainer" containerID="8bb69c2d9a359c701412949e83f446f7e8c8934bae695b2dfca8725dc709dd2d" Jul 14 18:29:09.443608 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:29:09.443592 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb69c2d9a359c701412949e83f446f7e8c8934bae695b2dfca8725dc709dd2d\": container with ID starting with 8bb69c2d9a359c701412949e83f446f7e8c8934bae695b2dfca8725dc709dd2d not found: ID does not exist" containerID="8bb69c2d9a359c701412949e83f446f7e8c8934bae695b2dfca8725dc709dd2d" Jul 14 18:29:09.443659 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:09.443616 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb69c2d9a359c701412949e83f446f7e8c8934bae695b2dfca8725dc709dd2d"} err="failed to get container status \"8bb69c2d9a359c701412949e83f446f7e8c8934bae695b2dfca8725dc709dd2d\": rpc error: code = NotFound desc = could not find container \"8bb69c2d9a359c701412949e83f446f7e8c8934bae695b2dfca8725dc709dd2d\": container with ID starting with 8bb69c2d9a359c701412949e83f446f7e8c8934bae695b2dfca8725dc709dd2d not found: ID does not exist" Jul 14 18:29:09.462242 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:09.462220 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8svfr"] Jul 14 18:29:09.474111 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:09.474092 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8svfr"] Jul 14 18:29:11.221749 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:11.221720 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4" path="/var/lib/kubelet/pods/fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4/volumes" Jul 14 18:29:15.983121 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:15.983082 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-jhn8w" Jul 14 18:29:15.983521 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:15.983149 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-jhn8w" Jul 14 18:29:16.006275 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:16.006237 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-jhn8w" Jul 14 18:29:16.493749 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:16.493725 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-jhn8w" Jul 14 18:29:34.298762 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:34.298726 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:35.162295 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:35.162259 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:38.681548 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:38.681518 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tjswq"] Jul 14 18:29:38.681982 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:38.681917 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4" containerName="registry-server" Jul 14 18:29:38.681982 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:38.681929 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4" containerName="registry-server" Jul 14 18:29:38.682056 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:38.682001 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fac2c5bf-e37c-49ba-8e61-e1eaaf5e18b4" containerName="registry-server" Jul 14 18:29:38.685202 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:38.685184 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tjswq" Jul 14 18:29:38.689379 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:38.689358 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-pf9dt\"" Jul 14 18:29:38.716569 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:38.716549 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tjswq"] Jul 14 18:29:38.836354 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:38.836330 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgslz\" (UniqueName: \"kubernetes.io/projected/3eeaba31-888b-46a4-9c65-38f28e914254-kube-api-access-tgslz\") pod \"authorino-operator-75cc8d5cb7-tjswq\" (UID: \"3eeaba31-888b-46a4-9c65-38f28e914254\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tjswq" Jul 14 18:29:38.937416 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:38.937345 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgslz\" (UniqueName: \"kubernetes.io/projected/3eeaba31-888b-46a4-9c65-38f28e914254-kube-api-access-tgslz\") pod \"authorino-operator-75cc8d5cb7-tjswq\" (UID: \"3eeaba31-888b-46a4-9c65-38f28e914254\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tjswq" Jul 14 18:29:38.979708 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:38.979678 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgslz\" (UniqueName: \"kubernetes.io/projected/3eeaba31-888b-46a4-9c65-38f28e914254-kube-api-access-tgslz\") pod \"authorino-operator-75cc8d5cb7-tjswq\" (UID: \"3eeaba31-888b-46a4-9c65-38f28e914254\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tjswq" Jul 14 18:29:39.000125 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:39.000101 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-pf9dt\"" Jul 14 18:29:39.005808 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:39.005792 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tjswq" Jul 14 18:29:39.280547 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:39.280520 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tjswq"] Jul 14 18:29:39.283154 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:29:39.283127 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eeaba31_888b_46a4_9c65_38f28e914254.slice/crio-3943961388b15cc4c09320aec30216fd27956d0ddc99485ea39888d819138a03 WatchSource:0}: Error finding container 3943961388b15cc4c09320aec30216fd27956d0ddc99485ea39888d819138a03: Status 404 returned error can't find the container with id 3943961388b15cc4c09320aec30216fd27956d0ddc99485ea39888d819138a03 Jul 14 18:29:39.590157 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:39.590072 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tjswq" event={"ID":"3eeaba31-888b-46a4-9c65-38f28e914254","Type":"ContainerStarted","Data":"3943961388b15cc4c09320aec30216fd27956d0ddc99485ea39888d819138a03"} Jul 14 18:29:39.746134 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:39.746103 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:41.228052 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:41.227999 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:41.861329 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:41.861280 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s95x5"] Jul 14 18:29:41.866076 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:41.866050 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s95x5" Jul 14 18:29:41.869256 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:41.869227 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8s49\" (UniqueName: \"kubernetes.io/projected/73919d62-68e8-4b94-9508-5495e52a1902-kube-api-access-w8s49\") pod \"dns-operator-controller-manager-864dd78c4c-s95x5\" (UID: \"73919d62-68e8-4b94-9508-5495e52a1902\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s95x5" Jul 14 18:29:41.880589 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:41.880566 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-j6g56\"" Jul 14 18:29:41.880723 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:41.880603 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 14 18:29:41.885399 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:41.885369 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s95x5"] Jul 14 18:29:41.970670 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:41.970641 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8s49\" (UniqueName: \"kubernetes.io/projected/73919d62-68e8-4b94-9508-5495e52a1902-kube-api-access-w8s49\") pod \"dns-operator-controller-manager-864dd78c4c-s95x5\" (UID: \"73919d62-68e8-4b94-9508-5495e52a1902\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s95x5" Jul 14 18:29:42.012525 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:42.012505 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8s49\" (UniqueName: \"kubernetes.io/projected/73919d62-68e8-4b94-9508-5495e52a1902-kube-api-access-w8s49\") pod \"dns-operator-controller-manager-864dd78c4c-s95x5\" (UID: \"73919d62-68e8-4b94-9508-5495e52a1902\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s95x5" Jul 14 18:29:42.180188 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:42.180089 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s95x5" Jul 14 18:29:42.339112 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:42.339086 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s95x5"] Jul 14 18:29:42.341046 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:29:42.341018 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73919d62_68e8_4b94_9508_5495e52a1902.slice/crio-9cd072870888a76dd17a7dd6ffdb0f7375ffb36468ae2ab4667cee4c080f7e82 WatchSource:0}: Error finding container 9cd072870888a76dd17a7dd6ffdb0f7375ffb36468ae2ab4667cee4c080f7e82: Status 404 returned error can't find the container with id 9cd072870888a76dd17a7dd6ffdb0f7375ffb36468ae2ab4667cee4c080f7e82 Jul 14 18:29:42.618162 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:42.618126 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tjswq" event={"ID":"3eeaba31-888b-46a4-9c65-38f28e914254","Type":"ContainerStarted","Data":"7dda327e16b2e5181808b751baa0a0c16a4c266726efb6fe5101efbfc7cc7842"} Jul 14 18:29:42.618392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:42.618326 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tjswq" Jul 14 18:29:42.619528 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:42.619498 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s95x5" event={"ID":"73919d62-68e8-4b94-9508-5495e52a1902","Type":"ContainerStarted","Data":"9cd072870888a76dd17a7dd6ffdb0f7375ffb36468ae2ab4667cee4c080f7e82"} Jul 14 18:29:42.687104 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:42.687040 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tjswq" podStartSLOduration=1.977108623 podStartE2EDuration="4.687023984s" totalImagesPullingTime="2.709915361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:29:42.685331679 +0000 UTC m=+484.222779461" watchObservedRunningTime="2026-07-14 18:29:42.687023984 +0000 UTC m=+484.224471773" Jul 14 18:29:42.945819 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:42.945759 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:43.257881 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:43.257797 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:45.192250 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:45.192216 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q"] Jul 14 18:29:45.196008 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:45.195990 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" Jul 14 18:29:45.204264 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:45.204200 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-gszkf\"" Jul 14 18:29:45.238875 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:45.238816 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q"] Jul 14 18:29:45.297463 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:45.297437 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpcwf\" (UniqueName: \"kubernetes.io/projected/60740b12-860c-4704-a842-fb73f2102b75-kube-api-access-zpcwf\") pod \"limitador-operator-controller-manager-859d7b78bd-z7p6q\" (UID: \"60740b12-860c-4704-a842-fb73f2102b75\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" Jul 14 18:29:45.398930 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:45.398895 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpcwf\" (UniqueName: \"kubernetes.io/projected/60740b12-860c-4704-a842-fb73f2102b75-kube-api-access-zpcwf\") pod \"limitador-operator-controller-manager-859d7b78bd-z7p6q\" (UID: \"60740b12-860c-4704-a842-fb73f2102b75\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" Jul 14 18:29:45.446406 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:45.446370 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpcwf\" (UniqueName: \"kubernetes.io/projected/60740b12-860c-4704-a842-fb73f2102b75-kube-api-access-zpcwf\") pod \"limitador-operator-controller-manager-859d7b78bd-z7p6q\" (UID: \"60740b12-860c-4704-a842-fb73f2102b75\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" Jul 14 18:29:45.506416 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:45.506350 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" Jul 14 18:29:45.643804 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:45.643763 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s95x5" event={"ID":"73919d62-68e8-4b94-9508-5495e52a1902","Type":"ContainerStarted","Data":"51a3644b83300b18310cdb95d5155861b58b46a5e266a2e9e7ed8f065c6fc499"} Jul 14 18:29:45.643947 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:45.643893 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s95x5" Jul 14 18:29:45.653756 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:45.653733 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:45.681254 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:45.681231 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q"] Jul 14 18:29:45.684155 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:29:45.684133 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60740b12_860c_4704_a842_fb73f2102b75.slice/crio-9652b4779fdd666e79303b72366a092573d6158a9a31cc8a29b6dba7138dfe74 WatchSource:0}: Error finding container 9652b4779fdd666e79303b72366a092573d6158a9a31cc8a29b6dba7138dfe74: Status 404 returned error can't find the container with id 9652b4779fdd666e79303b72366a092573d6158a9a31cc8a29b6dba7138dfe74 Jul 14 18:29:45.705248 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:45.705209 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s95x5" podStartSLOduration=2.082715806 podStartE2EDuration="4.705195537s" totalImagesPullingTime="2.622479731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:29:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:29:45.69503819 +0000 UTC m=+487.232485974" watchObservedRunningTime="2026-07-14 18:29:45.705195537 +0000 UTC m=+487.242643321" Jul 14 18:29:46.047610 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:46.047583 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:46.650224 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:46.650191 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" event={"ID":"60740b12-860c-4704-a842-fb73f2102b75","Type":"ContainerStarted","Data":"9652b4779fdd666e79303b72366a092573d6158a9a31cc8a29b6dba7138dfe74"} Jul 14 18:29:48.589886 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:48.589849 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:48.663980 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:48.663948 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" event={"ID":"60740b12-860c-4704-a842-fb73f2102b75","Type":"ContainerStarted","Data":"b208a05c4e2ee3562740e7e8a540381fc941a4f5af25307c1a49ce053fbc0768"} Jul 14 18:29:48.664156 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:48.664061 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" Jul 14 18:29:51.307204 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:51.307172 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:51.450378 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:51.450345 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:51.623153 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:51.623048 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:53.632328 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:53.632274 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tjswq" Jul 14 18:29:53.660121 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:53.660074 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" podStartSLOduration=5.911158602 podStartE2EDuration="8.660057029s" totalImagesPullingTime="2.748898427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:29:48.699805603 +0000 UTC m=+490.237253387" watchObservedRunningTime="2026-07-14 18:29:53.660057029 +0000 UTC m=+495.197504814" Jul 14 18:29:55.113513 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:55.113478 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:55.220267 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:55.220233 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:56.653156 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:56.653128 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s95x5" Jul 14 18:29:58.882666 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:58.882633 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:58.981952 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:58.981915 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:59.353674 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:59.353631 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:59.480151 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:59.480117 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:29:59.672251 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:29:59.672163 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" Jul 14 18:30:02.216103 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:02.216075 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:02.319623 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:02.319579 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:02.807429 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:02.807402 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:02.905472 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:02.905444 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:11.727763 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:11.727724 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx"] Jul 14 18:30:11.731544 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:11.731526 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx" Jul 14 18:30:11.738651 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:11.738628 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r57vd\"" Jul 14 18:30:11.759827 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:11.759803 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx"] Jul 14 18:30:11.835411 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:11.835385 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfmss\" (UniqueName: \"kubernetes.io/projected/c5850a29-d971-4449-b2cd-a0c2a9f90e62-kube-api-access-jfmss\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8skvx\" (UID: \"c5850a29-d971-4449-b2cd-a0c2a9f90e62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx" Jul 14 18:30:11.835552 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:11.835428 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5850a29-d971-4449-b2cd-a0c2a9f90e62-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8skvx\" (UID: \"c5850a29-d971-4449-b2cd-a0c2a9f90e62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx" Jul 14 18:30:11.936651 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:11.936624 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfmss\" (UniqueName: \"kubernetes.io/projected/c5850a29-d971-4449-b2cd-a0c2a9f90e62-kube-api-access-jfmss\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8skvx\" (UID: \"c5850a29-d971-4449-b2cd-a0c2a9f90e62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx" Jul 14 18:30:11.936776 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:11.936663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5850a29-d971-4449-b2cd-a0c2a9f90e62-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8skvx\" (UID: \"c5850a29-d971-4449-b2cd-a0c2a9f90e62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx" Jul 14 18:30:11.937070 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:11.937053 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5850a29-d971-4449-b2cd-a0c2a9f90e62-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8skvx\" (UID: \"c5850a29-d971-4449-b2cd-a0c2a9f90e62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx" Jul 14 18:30:11.949952 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:11.949925 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfmss\" (UniqueName: \"kubernetes.io/projected/c5850a29-d971-4449-b2cd-a0c2a9f90e62-kube-api-access-jfmss\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8skvx\" (UID: \"c5850a29-d971-4449-b2cd-a0c2a9f90e62\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx" Jul 14 18:30:12.041894 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.041821 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx" Jul 14 18:30:12.182099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.182076 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx"] Jul 14 18:30:12.184059 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:30:12.184022 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5850a29_d971_4449_b2cd_a0c2a9f90e62.slice/crio-1e189e961665251d532fd7e21e8b381b99231a80805784c32cb69e64aa7c665b WatchSource:0}: Error finding container 1e189e961665251d532fd7e21e8b381b99231a80805784c32cb69e64aa7c665b: Status 404 returned error can't find the container with id 1e189e961665251d532fd7e21e8b381b99231a80805784c32cb69e64aa7c665b Jul 14 18:30:12.538462 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.538425 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx"] Jul 14 18:30:12.545967 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.545943 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx"] Jul 14 18:30:12.578301 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.578264 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd"] Jul 14 18:30:12.584929 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.584909 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" Jul 14 18:30:12.599193 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.599164 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q"] Jul 14 18:30:12.600581 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.599380 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" podUID="60740b12-860c-4704-a842-fb73f2102b75" containerName="manager" containerID="cri-o://b208a05c4e2ee3562740e7e8a540381fc941a4f5af25307c1a49ce053fbc0768" gracePeriod=2 Jul 14 18:30:12.602877 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.602854 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd"] Jul 14 18:30:12.608038 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.608012 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q"] Jul 14 18:30:12.632408 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:30:12.632370 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z7p6q\" is forbidden: User \"system:node:ip-10-0-132-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-224.ec2.internal' and this object" podUID="60740b12-860c-4704-a842-fb73f2102b75" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" Jul 14 18:30:12.652531 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.652510 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p2j9n"] Jul 14 18:30:12.653046 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.652975 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60740b12-860c-4704-a842-fb73f2102b75" containerName="manager" Jul 14 18:30:12.653046 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.653045 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="60740b12-860c-4704-a842-fb73f2102b75" containerName="manager" Jul 14 18:30:12.653640 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.653184 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="60740b12-860c-4704-a842-fb73f2102b75" containerName="manager" Jul 14 18:30:12.658540 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.658520 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p2j9n" Jul 14 18:30:12.661059 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:30:12.661033 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z7p6q\" is forbidden: User \"system:node:ip-10-0-132-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-224.ec2.internal' and this object" podUID="60740b12-860c-4704-a842-fb73f2102b75" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" Jul 14 18:30:12.676833 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.676810 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p2j9n"] Jul 14 18:30:12.745065 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.745035 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s67vh\" (UniqueName: \"kubernetes.io/projected/fb25b1f0-68d0-410e-9959-66757fffd214-kube-api-access-s67vh\") pod \"limitador-operator-controller-manager-859d7b78bd-p2j9n\" (UID: \"fb25b1f0-68d0-410e-9959-66757fffd214\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p2j9n" Jul 14 18:30:12.745394 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.745087 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9nb6\" (UniqueName: \"kubernetes.io/projected/5da87725-5c30-423c-b570-2cd6fb89c130-kube-api-access-l9nb6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-q4xnd\" (UID: \"5da87725-5c30-423c-b570-2cd6fb89c130\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" Jul 14 18:30:12.745394 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.745174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5da87725-5c30-423c-b570-2cd6fb89c130-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-q4xnd\" (UID: \"5da87725-5c30-423c-b570-2cd6fb89c130\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" Jul 14 18:30:12.804009 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.803943 2573 generic.go:356] "Generic (PLEG): container finished" podID="60740b12-860c-4704-a842-fb73f2102b75" containerID="b208a05c4e2ee3562740e7e8a540381fc941a4f5af25307c1a49ce053fbc0768" exitCode=0 Jul 14 18:30:12.836035 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.836008 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" Jul 14 18:30:12.838930 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:30:12.838907 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-z7p6q\" is forbidden: User \"system:node:ip-10-0-132-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-224.ec2.internal' and this object" podUID="60740b12-860c-4704-a842-fb73f2102b75" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" Jul 14 18:30:12.845837 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.845817 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nb6\" (UniqueName: \"kubernetes.io/projected/5da87725-5c30-423c-b570-2cd6fb89c130-kube-api-access-l9nb6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-q4xnd\" (UID: \"5da87725-5c30-423c-b570-2cd6fb89c130\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" Jul 14 18:30:12.845905 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.845891 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5da87725-5c30-423c-b570-2cd6fb89c130-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-q4xnd\" (UID: \"5da87725-5c30-423c-b570-2cd6fb89c130\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" Jul 14 18:30:12.845948 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.845923 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s67vh\" (UniqueName: \"kubernetes.io/projected/fb25b1f0-68d0-410e-9959-66757fffd214-kube-api-access-s67vh\") pod \"limitador-operator-controller-manager-859d7b78bd-p2j9n\" (UID: \"fb25b1f0-68d0-410e-9959-66757fffd214\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p2j9n" Jul 14 18:30:12.846312 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.846263 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5da87725-5c30-423c-b570-2cd6fb89c130-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-q4xnd\" (UID: \"5da87725-5c30-423c-b570-2cd6fb89c130\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" Jul 14 18:30:12.864543 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.864523 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9nb6\" (UniqueName: \"kubernetes.io/projected/5da87725-5c30-423c-b570-2cd6fb89c130-kube-api-access-l9nb6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-q4xnd\" (UID: \"5da87725-5c30-423c-b570-2cd6fb89c130\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" Jul 14 18:30:12.874560 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.874538 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s67vh\" (UniqueName: \"kubernetes.io/projected/fb25b1f0-68d0-410e-9959-66757fffd214-kube-api-access-s67vh\") pod \"limitador-operator-controller-manager-859d7b78bd-p2j9n\" (UID: \"fb25b1f0-68d0-410e-9959-66757fffd214\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p2j9n" Jul 14 18:30:12.898365 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.898343 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" Jul 14 18:30:12.947559 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.947135 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpcwf\" (UniqueName: \"kubernetes.io/projected/60740b12-860c-4704-a842-fb73f2102b75-kube-api-access-zpcwf\") pod \"60740b12-860c-4704-a842-fb73f2102b75\" (UID: \"60740b12-860c-4704-a842-fb73f2102b75\") " Jul 14 18:30:12.950488 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.949824 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60740b12-860c-4704-a842-fb73f2102b75-kube-api-access-zpcwf" (OuterVolumeSpecName: "kube-api-access-zpcwf") pod "60740b12-860c-4704-a842-fb73f2102b75" (UID: "60740b12-860c-4704-a842-fb73f2102b75"). InnerVolumeSpecName "kube-api-access-zpcwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:30:12.968930 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:12.968908 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p2j9n" Jul 14 18:30:13.049599 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:13.049560 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpcwf\" (UniqueName: \"kubernetes.io/projected/60740b12-860c-4704-a842-fb73f2102b75-kube-api-access-zpcwf\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:30:13.069737 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:13.069441 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd"] Jul 14 18:30:13.073950 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:30:13.073888 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da87725_5c30_423c_b570_2cd6fb89c130.slice/crio-74ec1611b05821de906cdedeced93697e315825f6f1c370e18cbeb6469e034b2 WatchSource:0}: Error finding container 74ec1611b05821de906cdedeced93697e315825f6f1c370e18cbeb6469e034b2: Status 404 returned error can't find the container with id 74ec1611b05821de906cdedeced93697e315825f6f1c370e18cbeb6469e034b2 Jul 14 18:30:13.074249 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:13.074196 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:13.144326 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:13.144299 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p2j9n"] Jul 14 18:30:13.146212 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:30:13.146177 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb25b1f0_68d0_410e_9959_66757fffd214.slice/crio-f15ef2801184576be54dec71c7a5577bb7f84a829b4c23f4b12e12cac808d63c WatchSource:0}: Error finding container f15ef2801184576be54dec71c7a5577bb7f84a829b4c23f4b12e12cac808d63c: Status 404 returned error can't find the container with id f15ef2801184576be54dec71c7a5577bb7f84a829b4c23f4b12e12cac808d63c Jul 14 18:30:13.222538 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:13.222508 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60740b12-860c-4704-a842-fb73f2102b75" path="/var/lib/kubelet/pods/60740b12-860c-4704-a842-fb73f2102b75/volumes" Jul 14 18:30:13.814438 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:13.814403 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" event={"ID":"5da87725-5c30-423c-b570-2cd6fb89c130","Type":"ContainerStarted","Data":"74ec1611b05821de906cdedeced93697e315825f6f1c370e18cbeb6469e034b2"} Jul 14 18:30:13.816152 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:13.816131 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-z7p6q" Jul 14 18:30:13.816274 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:13.816135 2573 scope.go:117] "RemoveContainer" containerID="b208a05c4e2ee3562740e7e8a540381fc941a4f5af25307c1a49ce053fbc0768" Jul 14 18:30:13.818401 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:13.818369 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p2j9n" event={"ID":"fb25b1f0-68d0-410e-9959-66757fffd214","Type":"ContainerStarted","Data":"393ba163b8152888c34939405308aa5b80664b1811b73c26bccc5b09d96878bb"} Jul 14 18:30:13.818780 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:13.818445 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p2j9n" event={"ID":"fb25b1f0-68d0-410e-9959-66757fffd214","Type":"ContainerStarted","Data":"f15ef2801184576be54dec71c7a5577bb7f84a829b4c23f4b12e12cac808d63c"} Jul 14 18:30:13.819217 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:13.819189 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p2j9n" Jul 14 18:30:13.916895 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:13.916828 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p2j9n" podStartSLOduration=1.916804558 podStartE2EDuration="1.916804558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:30:13.864973374 +0000 UTC m=+515.402421160" watchObservedRunningTime="2026-07-14 18:30:13.916804558 +0000 UTC m=+515.454252447" Jul 14 18:30:14.010863 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:14.010826 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:14.365961 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:14.365928 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:17.851244 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:17.851211 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" event={"ID":"5da87725-5c30-423c-b570-2cd6fb89c130","Type":"ContainerStarted","Data":"71afd6d9bc9f1d0e4c2f94c2dd04cc467f1260cc1b1f8d29d587a3cf1b43588f"} Jul 14 18:30:17.851676 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:17.851361 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" Jul 14 18:30:17.852988 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:17.852961 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx" podUID="c5850a29-d971-4449-b2cd-a0c2a9f90e62" containerName="manager" containerID="cri-o://f4948dcec69db891761dc920150ff465f089b86c6979fd66750d580f83463609" gracePeriod=2 Jul 14 18:30:17.932563 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:17.932518 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" podStartSLOduration=1.9578881780000001 podStartE2EDuration="5.932496266s" totalImagesPullingTime="3.974608088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:30:17.926691392 +0000 UTC m=+519.464139168" watchObservedRunningTime="2026-07-14 18:30:17.932496266 +0000 UTC m=+519.469944050" Jul 14 18:30:18.070797 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:18.070768 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:18.108260 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:18.108195 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx" Jul 14 18:30:18.120790 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:30:18.120746 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8skvx\" is forbidden: User \"system:node:ip-10-0-132-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-224.ec2.internal' and this object" podUID="c5850a29-d971-4449-b2cd-a0c2a9f90e62" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx" Jul 14 18:30:18.197028 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:18.196985 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5850a29-d971-4449-b2cd-a0c2a9f90e62-extensions-socket-volume\") pod \"c5850a29-d971-4449-b2cd-a0c2a9f90e62\" (UID: \"c5850a29-d971-4449-b2cd-a0c2a9f90e62\") " Jul 14 18:30:18.197222 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:18.197116 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfmss\" (UniqueName: \"kubernetes.io/projected/c5850a29-d971-4449-b2cd-a0c2a9f90e62-kube-api-access-jfmss\") pod \"c5850a29-d971-4449-b2cd-a0c2a9f90e62\" (UID: \"c5850a29-d971-4449-b2cd-a0c2a9f90e62\") " Jul 14 18:30:18.197337 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:18.197249 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5850a29-d971-4449-b2cd-a0c2a9f90e62-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c5850a29-d971-4449-b2cd-a0c2a9f90e62" (UID: "c5850a29-d971-4449-b2cd-a0c2a9f90e62"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:30:18.197513 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:18.197488 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5850a29-d971-4449-b2cd-a0c2a9f90e62-extensions-socket-volume\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:30:18.199201 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:18.199176 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5850a29-d971-4449-b2cd-a0c2a9f90e62-kube-api-access-jfmss" (OuterVolumeSpecName: "kube-api-access-jfmss") pod "c5850a29-d971-4449-b2cd-a0c2a9f90e62" (UID: "c5850a29-d971-4449-b2cd-a0c2a9f90e62"). InnerVolumeSpecName "kube-api-access-jfmss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:30:18.294627 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:18.294594 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:18.298889 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:18.298859 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfmss\" (UniqueName: \"kubernetes.io/projected/c5850a29-d971-4449-b2cd-a0c2a9f90e62-kube-api-access-jfmss\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:30:18.866858 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:18.866748 2573 generic.go:356] "Generic (PLEG): container finished" podID="c5850a29-d971-4449-b2cd-a0c2a9f90e62" containerID="f4948dcec69db891761dc920150ff465f089b86c6979fd66750d580f83463609" exitCode=2 Jul 14 18:30:18.866858 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:18.866787 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx" Jul 14 18:30:18.866858 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:18.866846 2573 scope.go:117] "RemoveContainer" containerID="f4948dcec69db891761dc920150ff465f089b86c6979fd66750d580f83463609" Jul 14 18:30:18.873949 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:30:18.873900 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8skvx\" is forbidden: User \"system:node:ip-10-0-132-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-224.ec2.internal' and this object" podUID="c5850a29-d971-4449-b2cd-a0c2a9f90e62" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx" Jul 14 18:30:18.876235 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:18.876217 2573 scope.go:117] "RemoveContainer" containerID="f4948dcec69db891761dc920150ff465f089b86c6979fd66750d580f83463609" Jul 14 18:30:18.876457 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:30:18.876436 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4948dcec69db891761dc920150ff465f089b86c6979fd66750d580f83463609\": container with ID starting with f4948dcec69db891761dc920150ff465f089b86c6979fd66750d580f83463609 not found: ID does not exist" containerID="f4948dcec69db891761dc920150ff465f089b86c6979fd66750d580f83463609" Jul 14 18:30:18.876553 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:18.876462 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4948dcec69db891761dc920150ff465f089b86c6979fd66750d580f83463609"} err="failed to get container status \"f4948dcec69db891761dc920150ff465f089b86c6979fd66750d580f83463609\": rpc error: code = NotFound desc = could not find container \"f4948dcec69db891761dc920150ff465f089b86c6979fd66750d580f83463609\": container with ID starting with f4948dcec69db891761dc920150ff465f089b86c6979fd66750d580f83463609 not found: ID does not exist" Jul 14 18:30:18.878748 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:30:18.878727 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8skvx\" is forbidden: User \"system:node:ip-10-0-132-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-224.ec2.internal' and this object" podUID="c5850a29-d971-4449-b2cd-a0c2a9f90e62" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx" Jul 14 18:30:19.220858 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:30:19.220818 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-8skvx\" is forbidden: User \"system:node:ip-10-0-132-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-224.ec2.internal' and this object" podUID="c5850a29-d971-4449-b2cd-a0c2a9f90e62" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8skvx" Jul 14 18:30:19.222244 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:19.222221 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5850a29-d971-4449-b2cd-a0c2a9f90e62" path="/var/lib/kubelet/pods/c5850a29-d971-4449-b2cd-a0c2a9f90e62/volumes" Jul 14 18:30:22.221002 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:22.220970 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:22.325132 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:22.325106 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:25.830544 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:25.830506 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:25.838763 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:25.838734 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p2j9n" Jul 14 18:30:25.925218 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:25.925189 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:28.869711 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:28.869678 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" Jul 14 18:30:30.433391 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:30.433345 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:40.918965 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:40.918930 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd"] Jul 14 18:30:40.919397 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:40.919137 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" podUID="5da87725-5c30-423c-b570-2cd6fb89c130" containerName="manager" containerID="cri-o://71afd6d9bc9f1d0e4c2f94c2dd04cc467f1260cc1b1f8d29d587a3cf1b43588f" gracePeriod=10 Jul 14 18:30:41.174025 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:41.173973 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" Jul 14 18:30:41.188905 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:41.188885 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5da87725-5c30-423c-b570-2cd6fb89c130-extensions-socket-volume\") pod \"5da87725-5c30-423c-b570-2cd6fb89c130\" (UID: \"5da87725-5c30-423c-b570-2cd6fb89c130\") " Jul 14 18:30:41.189036 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:41.189020 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9nb6\" (UniqueName: \"kubernetes.io/projected/5da87725-5c30-423c-b570-2cd6fb89c130-kube-api-access-l9nb6\") pod \"5da87725-5c30-423c-b570-2cd6fb89c130\" (UID: \"5da87725-5c30-423c-b570-2cd6fb89c130\") " Jul 14 18:30:41.189217 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:41.189190 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5da87725-5c30-423c-b570-2cd6fb89c130-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5da87725-5c30-423c-b570-2cd6fb89c130" (UID: "5da87725-5c30-423c-b570-2cd6fb89c130"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:30:41.189298 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:41.189265 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5da87725-5c30-423c-b570-2cd6fb89c130-extensions-socket-volume\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:30:41.190986 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:41.190966 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da87725-5c30-423c-b570-2cd6fb89c130-kube-api-access-l9nb6" (OuterVolumeSpecName: "kube-api-access-l9nb6") pod "5da87725-5c30-423c-b570-2cd6fb89c130" (UID: "5da87725-5c30-423c-b570-2cd6fb89c130"). InnerVolumeSpecName "kube-api-access-l9nb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:30:41.290510 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:41.290488 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9nb6\" (UniqueName: \"kubernetes.io/projected/5da87725-5c30-423c-b570-2cd6fb89c130-kube-api-access-l9nb6\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:30:42.004176 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:42.004139 2573 generic.go:356] "Generic (PLEG): container finished" podID="5da87725-5c30-423c-b570-2cd6fb89c130" containerID="71afd6d9bc9f1d0e4c2f94c2dd04cc467f1260cc1b1f8d29d587a3cf1b43588f" exitCode=0 Jul 14 18:30:42.004176 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:42.004181 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" Jul 14 18:30:42.004657 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:42.004226 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" event={"ID":"5da87725-5c30-423c-b570-2cd6fb89c130","Type":"ContainerDied","Data":"71afd6d9bc9f1d0e4c2f94c2dd04cc467f1260cc1b1f8d29d587a3cf1b43588f"} Jul 14 18:30:42.004657 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:42.004278 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd" event={"ID":"5da87725-5c30-423c-b570-2cd6fb89c130","Type":"ContainerDied","Data":"74ec1611b05821de906cdedeced93697e315825f6f1c370e18cbeb6469e034b2"} Jul 14 18:30:42.004657 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:42.004330 2573 scope.go:117] "RemoveContainer" containerID="71afd6d9bc9f1d0e4c2f94c2dd04cc467f1260cc1b1f8d29d587a3cf1b43588f" Jul 14 18:30:42.013051 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:42.013031 2573 scope.go:117] "RemoveContainer" containerID="71afd6d9bc9f1d0e4c2f94c2dd04cc467f1260cc1b1f8d29d587a3cf1b43588f" Jul 14 18:30:42.013339 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:30:42.013321 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71afd6d9bc9f1d0e4c2f94c2dd04cc467f1260cc1b1f8d29d587a3cf1b43588f\": container with ID starting with 71afd6d9bc9f1d0e4c2f94c2dd04cc467f1260cc1b1f8d29d587a3cf1b43588f not found: ID does not exist" containerID="71afd6d9bc9f1d0e4c2f94c2dd04cc467f1260cc1b1f8d29d587a3cf1b43588f" Jul 14 18:30:42.013378 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:42.013348 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71afd6d9bc9f1d0e4c2f94c2dd04cc467f1260cc1b1f8d29d587a3cf1b43588f"} err="failed to get container status \"71afd6d9bc9f1d0e4c2f94c2dd04cc467f1260cc1b1f8d29d587a3cf1b43588f\": rpc error: code = NotFound desc = could not find container \"71afd6d9bc9f1d0e4c2f94c2dd04cc467f1260cc1b1f8d29d587a3cf1b43588f\": container with ID starting with 71afd6d9bc9f1d0e4c2f94c2dd04cc467f1260cc1b1f8d29d587a3cf1b43588f not found: ID does not exist" Jul 14 18:30:42.035374 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:42.035349 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd"] Jul 14 18:30:42.041144 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:42.041123 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-q4xnd"] Jul 14 18:30:43.223043 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:43.223013 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5da87725-5c30-423c-b570-2cd6fb89c130" path="/var/lib/kubelet/pods/5da87725-5c30-423c-b570-2cd6fb89c130/volumes" Jul 14 18:30:45.525711 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.525682 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8"] Jul 14 18:30:45.526129 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.526116 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da87725-5c30-423c-b570-2cd6fb89c130" containerName="manager" Jul 14 18:30:45.526129 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.526130 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da87725-5c30-423c-b570-2cd6fb89c130" containerName="manager" Jul 14 18:30:45.526208 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.526140 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5850a29-d971-4449-b2cd-a0c2a9f90e62" containerName="manager" Jul 14 18:30:45.526208 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.526145 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5850a29-d971-4449-b2cd-a0c2a9f90e62" containerName="manager" Jul 14 18:30:45.526269 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.526216 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da87725-5c30-423c-b570-2cd6fb89c130" containerName="manager" Jul 14 18:30:45.526269 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.526225 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5850a29-d971-4449-b2cd-a0c2a9f90e62" containerName="manager" Jul 14 18:30:45.530771 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.530754 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.533772 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.533741 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-p2xw4\"" Jul 14 18:30:45.542520 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.542500 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8"] Jul 14 18:30:45.630029 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.629997 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83981a26-6460-417f-9442-602373954802-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.630274 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.630245 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83981a26-6460-417f-9442-602373954802-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.630429 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.630329 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwdrs\" (UniqueName: \"kubernetes.io/projected/83981a26-6460-417f-9442-602373954802-kube-api-access-bwdrs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.630429 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.630416 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83981a26-6460-417f-9442-602373954802-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.630551 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.630445 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83981a26-6460-417f-9442-602373954802-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.630551 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.630478 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83981a26-6460-417f-9442-602373954802-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.630551 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.630501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83981a26-6460-417f-9442-602373954802-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.630551 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.630533 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83981a26-6460-417f-9442-602373954802-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.630753 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.630587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83981a26-6460-417f-9442-602373954802-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.731669 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.731636 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83981a26-6460-417f-9442-602373954802-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.731669 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.731672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83981a26-6460-417f-9442-602373954802-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.731873 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.731689 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83981a26-6460-417f-9442-602373954802-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.731938 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.731890 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83981a26-6460-417f-9442-602373954802-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.732001 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.731949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83981a26-6460-417f-9442-602373954802-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.732058 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.732004 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83981a26-6460-417f-9442-602373954802-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.732058 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.732018 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83981a26-6460-417f-9442-602373954802-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.732173 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.732157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83981a26-6460-417f-9442-602373954802-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.732229 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.732205 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdrs\" (UniqueName: \"kubernetes.io/projected/83981a26-6460-417f-9442-602373954802-kube-api-access-bwdrs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.732306 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.732260 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83981a26-6460-417f-9442-602373954802-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.732306 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.732207 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83981a26-6460-417f-9442-602373954802-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.732421 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.732311 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83981a26-6460-417f-9442-602373954802-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.732483 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.732464 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83981a26-6460-417f-9442-602373954802-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.732537 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.732505 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83981a26-6460-417f-9442-602373954802-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.734321 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.734298 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83981a26-6460-417f-9442-602373954802-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.734461 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.734442 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83981a26-6460-417f-9442-602373954802-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.742259 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.742236 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83981a26-6460-417f-9442-602373954802-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.742513 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.742491 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdrs\" (UniqueName: \"kubernetes.io/projected/83981a26-6460-417f-9442-602373954802-kube-api-access-bwdrs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rlbm8\" (UID: \"83981a26-6460-417f-9442-602373954802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.841485 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.841422 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:45.989463 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.989436 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8"] Jul 14 18:30:45.991889 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:30:45.991867 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83981a26_6460_417f_9442_602373954802.slice/crio-78702e58d3827ea5a079d2809cc43725fbce6f7a34227bf7b85d5031e3437a08 WatchSource:0}: Error finding container 78702e58d3827ea5a079d2809cc43725fbce6f7a34227bf7b85d5031e3437a08: Status 404 returned error can't find the container with id 78702e58d3827ea5a079d2809cc43725fbce6f7a34227bf7b85d5031e3437a08 Jul 14 18:30:45.994519 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.994488 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:30:45.994601 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.994551 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:30:45.994601 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:45.994578 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:30:46.031098 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:46.031071 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" event={"ID":"83981a26-6460-417f-9442-602373954802","Type":"ContainerStarted","Data":"78702e58d3827ea5a079d2809cc43725fbce6f7a34227bf7b85d5031e3437a08"} Jul 14 18:30:46.041078 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:46.041057 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:46.141185 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:46.141054 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:47.037714 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:47.037673 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" event={"ID":"83981a26-6460-417f-9442-602373954802","Type":"ContainerStarted","Data":"7d6ad8f02e97a8d80c6f30dd5cb1ef2d80ede30029d8dde19696a943bc268031"} Jul 14 18:30:47.071335 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:47.071275 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" podStartSLOduration=2.071259231 podStartE2EDuration="2.071259231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:30:47.068979005 +0000 UTC m=+548.606426790" watchObservedRunningTime="2026-07-14 18:30:47.071259231 +0000 UTC m=+548.608707015" Jul 14 18:30:47.842156 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:47.842121 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:47.846626 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:47.846600 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:48.043443 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:48.043414 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:48.044438 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:48.044419 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:30:52.196682 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:52.196649 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:54.741000 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:54.740961 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:30:54.839025 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:30:54.838988 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:00.122941 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.122906 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:00.139241 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.139207 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rn7p4"] Jul 14 18:31:00.142607 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.142587 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" Jul 14 18:31:00.146103 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.146085 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4g49d\"" Jul 14 18:31:00.146226 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.146111 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 14 18:31:00.170081 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.170036 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rn7p4"] Jul 14 18:31:00.260497 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.260466 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/94cd179a-b991-405a-9b95-ddcd363a29d9-config-file\") pod \"limitador-limitador-64c94fd64f-rn7p4\" (UID: \"94cd179a-b991-405a-9b95-ddcd363a29d9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" Jul 14 18:31:00.260662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.260596 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct7k4\" (UniqueName: \"kubernetes.io/projected/94cd179a-b991-405a-9b95-ddcd363a29d9-kube-api-access-ct7k4\") pod \"limitador-limitador-64c94fd64f-rn7p4\" (UID: \"94cd179a-b991-405a-9b95-ddcd363a29d9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" Jul 14 18:31:00.275328 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.275277 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rn7p4"] Jul 14 18:31:00.305463 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.305433 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:00.361446 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.361410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct7k4\" (UniqueName: \"kubernetes.io/projected/94cd179a-b991-405a-9b95-ddcd363a29d9-kube-api-access-ct7k4\") pod \"limitador-limitador-64c94fd64f-rn7p4\" (UID: \"94cd179a-b991-405a-9b95-ddcd363a29d9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" Jul 14 18:31:00.361622 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.361487 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/94cd179a-b991-405a-9b95-ddcd363a29d9-config-file\") pod \"limitador-limitador-64c94fd64f-rn7p4\" (UID: \"94cd179a-b991-405a-9b95-ddcd363a29d9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" Jul 14 18:31:00.362198 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.362178 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/94cd179a-b991-405a-9b95-ddcd363a29d9-config-file\") pod \"limitador-limitador-64c94fd64f-rn7p4\" (UID: \"94cd179a-b991-405a-9b95-ddcd363a29d9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" Jul 14 18:31:00.382674 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.382606 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct7k4\" (UniqueName: \"kubernetes.io/projected/94cd179a-b991-405a-9b95-ddcd363a29d9-kube-api-access-ct7k4\") pod \"limitador-limitador-64c94fd64f-rn7p4\" (UID: \"94cd179a-b991-405a-9b95-ddcd363a29d9\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" Jul 14 18:31:00.456096 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.456056 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" Jul 14 18:31:00.470935 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.470888 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:00.607997 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:00.607969 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rn7p4"] Jul 14 18:31:00.609363 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:31:00.609337 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94cd179a_b991_405a_9b95_ddcd363a29d9.slice/crio-9b721e0d723d1638264b148799ea677efa12a4ec36c893650ebbcbc2f034b55f WatchSource:0}: Error finding container 9b721e0d723d1638264b148799ea677efa12a4ec36c893650ebbcbc2f034b55f: Status 404 returned error can't find the container with id 9b721e0d723d1638264b148799ea677efa12a4ec36c893650ebbcbc2f034b55f Jul 14 18:31:01.111120 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:01.111086 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" event={"ID":"94cd179a-b991-405a-9b95-ddcd363a29d9","Type":"ContainerStarted","Data":"9b721e0d723d1638264b148799ea677efa12a4ec36c893650ebbcbc2f034b55f"} Jul 14 18:31:01.265496 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:01.265460 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:01.417509 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:01.417419 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:01.587875 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:01.587248 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-nn9pd"] Jul 14 18:31:01.592615 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:01.592594 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nn9pd" Jul 14 18:31:01.597020 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:01.596984 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-z7rpx\"" Jul 14 18:31:01.627021 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:01.626964 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nn9pd"] Jul 14 18:31:01.674834 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:01.674607 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cdkm\" (UniqueName: \"kubernetes.io/projected/a872d741-5c2f-414a-9760-613bb4809e14-kube-api-access-9cdkm\") pod \"authorino-75d556bd77-nn9pd\" (UID: \"a872d741-5c2f-414a-9760-613bb4809e14\") " pod="kuadrant-system/authorino-75d556bd77-nn9pd" Jul 14 18:31:01.775548 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:01.775509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cdkm\" (UniqueName: \"kubernetes.io/projected/a872d741-5c2f-414a-9760-613bb4809e14-kube-api-access-9cdkm\") pod \"authorino-75d556bd77-nn9pd\" (UID: \"a872d741-5c2f-414a-9760-613bb4809e14\") " pod="kuadrant-system/authorino-75d556bd77-nn9pd" Jul 14 18:31:01.808134 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:01.808099 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cdkm\" (UniqueName: \"kubernetes.io/projected/a872d741-5c2f-414a-9760-613bb4809e14-kube-api-access-9cdkm\") pod \"authorino-75d556bd77-nn9pd\" (UID: \"a872d741-5c2f-414a-9760-613bb4809e14\") " pod="kuadrant-system/authorino-75d556bd77-nn9pd" Jul 14 18:31:01.909349 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:01.909312 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nn9pd" Jul 14 18:31:02.098748 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:02.098713 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nn9pd"] Jul 14 18:31:02.913687 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:31:02.913654 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda872d741_5c2f_414a_9760_613bb4809e14.slice/crio-12b637bd9906a17f00d97b15c4c6d133573b1fc8d5593530483e66482e794122 WatchSource:0}: Error finding container 12b637bd9906a17f00d97b15c4c6d133573b1fc8d5593530483e66482e794122: Status 404 returned error can't find the container with id 12b637bd9906a17f00d97b15c4c6d133573b1fc8d5593530483e66482e794122 Jul 14 18:31:03.125121 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:03.125086 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nn9pd" event={"ID":"a872d741-5c2f-414a-9760-613bb4809e14","Type":"ContainerStarted","Data":"12b637bd9906a17f00d97b15c4c6d133573b1fc8d5593530483e66482e794122"} Jul 14 18:31:04.133840 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:04.133800 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" event={"ID":"94cd179a-b991-405a-9b95-ddcd363a29d9","Type":"ContainerStarted","Data":"2e3eb1a2c01362ae11d3d2eb6eb87e2f79388a4bc366f3960142c95fc2313878"} Jul 14 18:31:04.134329 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:04.133919 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" Jul 14 18:31:06.147392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:06.147308 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nn9pd" event={"ID":"a872d741-5c2f-414a-9760-613bb4809e14","Type":"ContainerStarted","Data":"f7617b2a886767c194914695987287b84442e0fb8af0f5e3432109879f2b52be"} Jul 14 18:31:06.171344 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:06.171297 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-nn9pd" podStartSLOduration=2.203836834 podStartE2EDuration="5.171264992s" totalImagesPullingTime="2.967428158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:31:06.169203014 +0000 UTC m=+567.706650798" watchObservedRunningTime="2026-07-14 18:31:06.171264992 +0000 UTC m=+567.708712776" Jul 14 18:31:06.172167 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:06.172140 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" podStartSLOduration=3.335007117 podStartE2EDuration="6.172130207s" totalImagesPullingTime="2.83712309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:31:04.163325228 +0000 UTC m=+565.700773012" watchObservedRunningTime="2026-07-14 18:31:06.172130207 +0000 UTC m=+567.709577991" Jul 14 18:31:15.141364 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:15.141332 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" Jul 14 18:31:17.885128 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:17.885096 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rn7p4"] Jul 14 18:31:17.885528 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:17.885327 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" podUID="94cd179a-b991-405a-9b95-ddcd363a29d9" containerName="limitador" containerID="cri-o://2e3eb1a2c01362ae11d3d2eb6eb87e2f79388a4bc366f3960142c95fc2313878" gracePeriod=30 Jul 14 18:31:18.437408 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:18.437384 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" Jul 14 18:31:18.535431 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:18.535401 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct7k4\" (UniqueName: \"kubernetes.io/projected/94cd179a-b991-405a-9b95-ddcd363a29d9-kube-api-access-ct7k4\") pod \"94cd179a-b991-405a-9b95-ddcd363a29d9\" (UID: \"94cd179a-b991-405a-9b95-ddcd363a29d9\") " Jul 14 18:31:18.535561 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:18.535489 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/94cd179a-b991-405a-9b95-ddcd363a29d9-config-file\") pod \"94cd179a-b991-405a-9b95-ddcd363a29d9\" (UID: \"94cd179a-b991-405a-9b95-ddcd363a29d9\") " Jul 14 18:31:18.535839 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:18.535808 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94cd179a-b991-405a-9b95-ddcd363a29d9-config-file" (OuterVolumeSpecName: "config-file") pod "94cd179a-b991-405a-9b95-ddcd363a29d9" (UID: "94cd179a-b991-405a-9b95-ddcd363a29d9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:31:18.537416 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:18.537394 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94cd179a-b991-405a-9b95-ddcd363a29d9-kube-api-access-ct7k4" (OuterVolumeSpecName: "kube-api-access-ct7k4") pod "94cd179a-b991-405a-9b95-ddcd363a29d9" (UID: "94cd179a-b991-405a-9b95-ddcd363a29d9"). InnerVolumeSpecName "kube-api-access-ct7k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:31:18.637085 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:18.637052 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct7k4\" (UniqueName: \"kubernetes.io/projected/94cd179a-b991-405a-9b95-ddcd363a29d9-kube-api-access-ct7k4\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:31:18.637085 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:18.637080 2573 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/94cd179a-b991-405a-9b95-ddcd363a29d9-config-file\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:31:19.223471 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:19.223431 2573 generic.go:356] "Generic (PLEG): container finished" podID="94cd179a-b991-405a-9b95-ddcd363a29d9" containerID="2e3eb1a2c01362ae11d3d2eb6eb87e2f79388a4bc366f3960142c95fc2313878" exitCode=0 Jul 14 18:31:19.223942 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:19.223468 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" event={"ID":"94cd179a-b991-405a-9b95-ddcd363a29d9","Type":"ContainerDied","Data":"2e3eb1a2c01362ae11d3d2eb6eb87e2f79388a4bc366f3960142c95fc2313878"} Jul 14 18:31:19.223942 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:19.223532 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" event={"ID":"94cd179a-b991-405a-9b95-ddcd363a29d9","Type":"ContainerDied","Data":"9b721e0d723d1638264b148799ea677efa12a4ec36c893650ebbcbc2f034b55f"} Jul 14 18:31:19.223942 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:19.223450 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rn7p4" Jul 14 18:31:19.223942 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:19.223565 2573 scope.go:117] "RemoveContainer" containerID="2e3eb1a2c01362ae11d3d2eb6eb87e2f79388a4bc366f3960142c95fc2313878" Jul 14 18:31:19.233356 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:19.233339 2573 scope.go:117] "RemoveContainer" containerID="2e3eb1a2c01362ae11d3d2eb6eb87e2f79388a4bc366f3960142c95fc2313878" Jul 14 18:31:19.233607 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:31:19.233584 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e3eb1a2c01362ae11d3d2eb6eb87e2f79388a4bc366f3960142c95fc2313878\": container with ID starting with 2e3eb1a2c01362ae11d3d2eb6eb87e2f79388a4bc366f3960142c95fc2313878 not found: ID does not exist" containerID="2e3eb1a2c01362ae11d3d2eb6eb87e2f79388a4bc366f3960142c95fc2313878" Jul 14 18:31:19.233666 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:19.233620 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e3eb1a2c01362ae11d3d2eb6eb87e2f79388a4bc366f3960142c95fc2313878"} err="failed to get container status \"2e3eb1a2c01362ae11d3d2eb6eb87e2f79388a4bc366f3960142c95fc2313878\": rpc error: code = NotFound desc = could not find container \"2e3eb1a2c01362ae11d3d2eb6eb87e2f79388a4bc366f3960142c95fc2313878\": container with ID starting with 2e3eb1a2c01362ae11d3d2eb6eb87e2f79388a4bc366f3960142c95fc2313878 not found: ID does not exist" Jul 14 18:31:19.254513 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:19.254490 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rn7p4"] Jul 14 18:31:19.263778 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:19.263757 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rn7p4"] Jul 14 18:31:20.016311 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.016254 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:20.180214 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.180185 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:20.457939 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.457904 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hwt49"] Jul 14 18:31:20.458367 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.458353 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94cd179a-b991-405a-9b95-ddcd363a29d9" containerName="limitador" Jul 14 18:31:20.458367 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.458369 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="94cd179a-b991-405a-9b95-ddcd363a29d9" containerName="limitador" Jul 14 18:31:20.458460 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.458446 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="94cd179a-b991-405a-9b95-ddcd363a29d9" containerName="limitador" Jul 14 18:31:20.462978 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.462958 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwt49" Jul 14 18:31:20.465987 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.465967 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-gr6gq\"" Jul 14 18:31:20.466106 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.465968 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 14 18:31:20.467557 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.467540 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 14 18:31:20.467686 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.467559 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:31:20.473617 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.473594 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hwt49"] Jul 14 18:31:20.498234 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.498194 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:20.654494 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.654459 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2fdcd8d1-06ca-42f9-a8c8-14d646614ec4-data\") pod \"postgres-69fb4594cf-hwt49\" (UID: \"2fdcd8d1-06ca-42f9-a8c8-14d646614ec4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwt49" Jul 14 18:31:20.654656 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.654521 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f98kh\" (UniqueName: \"kubernetes.io/projected/2fdcd8d1-06ca-42f9-a8c8-14d646614ec4-kube-api-access-f98kh\") pod \"postgres-69fb4594cf-hwt49\" (UID: \"2fdcd8d1-06ca-42f9-a8c8-14d646614ec4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwt49" Jul 14 18:31:20.755492 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.755415 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2fdcd8d1-06ca-42f9-a8c8-14d646614ec4-data\") pod \"postgres-69fb4594cf-hwt49\" (UID: \"2fdcd8d1-06ca-42f9-a8c8-14d646614ec4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwt49" Jul 14 18:31:20.755492 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.755471 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f98kh\" (UniqueName: \"kubernetes.io/projected/2fdcd8d1-06ca-42f9-a8c8-14d646614ec4-kube-api-access-f98kh\") pod \"postgres-69fb4594cf-hwt49\" (UID: \"2fdcd8d1-06ca-42f9-a8c8-14d646614ec4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwt49" Jul 14 18:31:20.755774 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.755756 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2fdcd8d1-06ca-42f9-a8c8-14d646614ec4-data\") pod \"postgres-69fb4594cf-hwt49\" (UID: \"2fdcd8d1-06ca-42f9-a8c8-14d646614ec4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwt49" Jul 14 18:31:20.778757 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:20.778726 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f98kh\" (UniqueName: \"kubernetes.io/projected/2fdcd8d1-06ca-42f9-a8c8-14d646614ec4-kube-api-access-f98kh\") pod \"postgres-69fb4594cf-hwt49\" (UID: \"2fdcd8d1-06ca-42f9-a8c8-14d646614ec4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwt49" Jul 14 18:31:21.073372 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:21.073271 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwt49" Jul 14 18:31:21.222964 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:31:21.222924 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdcd8d1_06ca_42f9_a8c8_14d646614ec4.slice/crio-8e374a0282af794f7cf6f17a2e07183b8a408eb964cf72d0021093b39c1f9384 WatchSource:0}: Error finding container 8e374a0282af794f7cf6f17a2e07183b8a408eb964cf72d0021093b39c1f9384: Status 404 returned error can't find the container with id 8e374a0282af794f7cf6f17a2e07183b8a408eb964cf72d0021093b39c1f9384 Jul 14 18:31:21.224482 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:21.224452 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94cd179a-b991-405a-9b95-ddcd363a29d9" path="/var/lib/kubelet/pods/94cd179a-b991-405a-9b95-ddcd363a29d9/volumes" Jul 14 18:31:21.224863 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:21.224782 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hwt49"] Jul 14 18:31:21.236363 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:21.236335 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwt49" event={"ID":"2fdcd8d1-06ca-42f9-a8c8-14d646614ec4","Type":"ContainerStarted","Data":"8e374a0282af794f7cf6f17a2e07183b8a408eb964cf72d0021093b39c1f9384"} Jul 14 18:31:27.274384 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:27.274345 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwt49" event={"ID":"2fdcd8d1-06ca-42f9-a8c8-14d646614ec4","Type":"ContainerStarted","Data":"2adae54e0b2fe364d1e51cb99fab249837eab70a1b986c5c872fff7d3e548f0d"} Jul 14 18:31:27.274750 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:27.274427 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwt49" Jul 14 18:31:27.306361 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:27.306315 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwt49" podStartSLOduration=2.050786699 podStartE2EDuration="7.306300802s" totalImagesPullingTime="5.255514103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:31:27.3041167 +0000 UTC m=+588.841564486" watchObservedRunningTime="2026-07-14 18:31:27.306300802 +0000 UTC m=+588.843748578" Jul 14 18:31:33.310183 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:33.310155 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hwt49" Jul 14 18:31:33.749729 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:33.749699 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:34.052123 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:34.052036 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:34.761413 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:34.761378 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55dcc8665-gw4fn"] Jul 14 18:31:34.765485 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:34.765457 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55dcc8665-gw4fn" Jul 14 18:31:34.771791 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:34.771761 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 14 18:31:34.781416 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:34.781393 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55dcc8665-gw4fn"] Jul 14 18:31:34.877076 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:34.877040 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a033555e-12ba-4a20-9358-0e122a1d0cfc-tls-cert\") pod \"authorino-55dcc8665-gw4fn\" (UID: \"a033555e-12ba-4a20-9358-0e122a1d0cfc\") " pod="kuadrant-system/authorino-55dcc8665-gw4fn" Jul 14 18:31:34.877218 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:34.877156 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l668t\" (UniqueName: \"kubernetes.io/projected/a033555e-12ba-4a20-9358-0e122a1d0cfc-kube-api-access-l668t\") pod \"authorino-55dcc8665-gw4fn\" (UID: \"a033555e-12ba-4a20-9358-0e122a1d0cfc\") " pod="kuadrant-system/authorino-55dcc8665-gw4fn" Jul 14 18:31:34.978573 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:34.978542 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l668t\" (UniqueName: \"kubernetes.io/projected/a033555e-12ba-4a20-9358-0e122a1d0cfc-kube-api-access-l668t\") pod \"authorino-55dcc8665-gw4fn\" (UID: \"a033555e-12ba-4a20-9358-0e122a1d0cfc\") " pod="kuadrant-system/authorino-55dcc8665-gw4fn" Jul 14 18:31:34.978718 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:34.978595 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a033555e-12ba-4a20-9358-0e122a1d0cfc-tls-cert\") pod \"authorino-55dcc8665-gw4fn\" (UID: \"a033555e-12ba-4a20-9358-0e122a1d0cfc\") " pod="kuadrant-system/authorino-55dcc8665-gw4fn" Jul 14 18:31:34.980923 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:34.980895 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a033555e-12ba-4a20-9358-0e122a1d0cfc-tls-cert\") pod \"authorino-55dcc8665-gw4fn\" (UID: \"a033555e-12ba-4a20-9358-0e122a1d0cfc\") " pod="kuadrant-system/authorino-55dcc8665-gw4fn" Jul 14 18:31:34.999737 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:34.999711 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l668t\" (UniqueName: \"kubernetes.io/projected/a033555e-12ba-4a20-9358-0e122a1d0cfc-kube-api-access-l668t\") pod \"authorino-55dcc8665-gw4fn\" (UID: \"a033555e-12ba-4a20-9358-0e122a1d0cfc\") " pod="kuadrant-system/authorino-55dcc8665-gw4fn" Jul 14 18:31:35.074918 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:35.074834 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55dcc8665-gw4fn" Jul 14 18:31:35.228997 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:35.228973 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55dcc8665-gw4fn"] Jul 14 18:31:35.230359 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:31:35.230323 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda033555e_12ba_4a20_9358_0e122a1d0cfc.slice/crio-f2b3050ec219ae5c3647933c6958bfa823bad9c6ceaf3d142fac6351628e7faa WatchSource:0}: Error finding container f2b3050ec219ae5c3647933c6958bfa823bad9c6ceaf3d142fac6351628e7faa: Status 404 returned error can't find the container with id f2b3050ec219ae5c3647933c6958bfa823bad9c6ceaf3d142fac6351628e7faa Jul 14 18:31:35.323985 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:35.323958 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55dcc8665-gw4fn" event={"ID":"a033555e-12ba-4a20-9358-0e122a1d0cfc","Type":"ContainerStarted","Data":"f2b3050ec219ae5c3647933c6958bfa823bad9c6ceaf3d142fac6351628e7faa"} Jul 14 18:31:36.331401 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:36.331368 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55dcc8665-gw4fn" event={"ID":"a033555e-12ba-4a20-9358-0e122a1d0cfc","Type":"ContainerStarted","Data":"185dc92d430f3529fd731de2ff17b120666151ba30cc6fbdb78c578bb025c91e"} Jul 14 18:31:36.362774 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:36.362727 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55dcc8665-gw4fn" podStartSLOduration=1.985593943 podStartE2EDuration="2.362712033s" totalImagesPullingTime="377.11809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:31:36.362311387 +0000 UTC m=+597.899759168" watchObservedRunningTime="2026-07-14 18:31:36.362712033 +0000 UTC m=+597.900159818" Jul 14 18:31:36.412775 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:36.412741 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nn9pd"] Jul 14 18:31:36.412967 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:36.412941 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-nn9pd" podUID="a872d741-5c2f-414a-9760-613bb4809e14" containerName="authorino" containerID="cri-o://f7617b2a886767c194914695987287b84442e0fb8af0f5e3432109879f2b52be" gracePeriod=30 Jul 14 18:31:36.667560 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:36.667534 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nn9pd" Jul 14 18:31:36.693046 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:36.693021 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cdkm\" (UniqueName: \"kubernetes.io/projected/a872d741-5c2f-414a-9760-613bb4809e14-kube-api-access-9cdkm\") pod \"a872d741-5c2f-414a-9760-613bb4809e14\" (UID: \"a872d741-5c2f-414a-9760-613bb4809e14\") " Jul 14 18:31:36.695068 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:36.695034 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a872d741-5c2f-414a-9760-613bb4809e14-kube-api-access-9cdkm" (OuterVolumeSpecName: "kube-api-access-9cdkm") pod "a872d741-5c2f-414a-9760-613bb4809e14" (UID: "a872d741-5c2f-414a-9760-613bb4809e14"). InnerVolumeSpecName "kube-api-access-9cdkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:31:36.793880 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:36.793852 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cdkm\" (UniqueName: \"kubernetes.io/projected/a872d741-5c2f-414a-9760-613bb4809e14-kube-api-access-9cdkm\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:31:37.338366 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:37.338333 2573 generic.go:356] "Generic (PLEG): container finished" podID="a872d741-5c2f-414a-9760-613bb4809e14" containerID="f7617b2a886767c194914695987287b84442e0fb8af0f5e3432109879f2b52be" exitCode=0 Jul 14 18:31:37.338723 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:37.338380 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nn9pd" Jul 14 18:31:37.338723 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:37.338421 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nn9pd" event={"ID":"a872d741-5c2f-414a-9760-613bb4809e14","Type":"ContainerDied","Data":"f7617b2a886767c194914695987287b84442e0fb8af0f5e3432109879f2b52be"} Jul 14 18:31:37.338723 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:37.338463 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nn9pd" event={"ID":"a872d741-5c2f-414a-9760-613bb4809e14","Type":"ContainerDied","Data":"12b637bd9906a17f00d97b15c4c6d133573b1fc8d5593530483e66482e794122"} Jul 14 18:31:37.338723 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:37.338486 2573 scope.go:117] "RemoveContainer" containerID="f7617b2a886767c194914695987287b84442e0fb8af0f5e3432109879f2b52be" Jul 14 18:31:37.348258 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:37.348225 2573 scope.go:117] "RemoveContainer" containerID="f7617b2a886767c194914695987287b84442e0fb8af0f5e3432109879f2b52be" Jul 14 18:31:37.348537 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:31:37.348517 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7617b2a886767c194914695987287b84442e0fb8af0f5e3432109879f2b52be\": container with ID starting with f7617b2a886767c194914695987287b84442e0fb8af0f5e3432109879f2b52be not found: ID does not exist" containerID="f7617b2a886767c194914695987287b84442e0fb8af0f5e3432109879f2b52be" Jul 14 18:31:37.348603 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:37.348546 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7617b2a886767c194914695987287b84442e0fb8af0f5e3432109879f2b52be"} err="failed to get container status \"f7617b2a886767c194914695987287b84442e0fb8af0f5e3432109879f2b52be\": rpc error: code = NotFound desc = could not find container \"f7617b2a886767c194914695987287b84442e0fb8af0f5e3432109879f2b52be\": container with ID starting with f7617b2a886767c194914695987287b84442e0fb8af0f5e3432109879f2b52be not found: ID does not exist" Jul 14 18:31:37.385542 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:37.385515 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nn9pd"] Jul 14 18:31:37.394764 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:37.394745 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nn9pd"] Jul 14 18:31:38.843280 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:38.843245 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:31:38.843695 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:38.843278 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:31:38.850864 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:38.850844 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:31:38.851087 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:38.851071 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:31:39.222326 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:39.222276 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a872d741-5c2f-414a-9760-613bb4809e14" path="/var/lib/kubelet/pods/a872d741-5c2f-414a-9760-613bb4809e14/volumes" Jul 14 18:31:41.882060 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:41.882024 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:41.919514 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:41.919489 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cb7ccd7b5-hqbgr"] Jul 14 18:31:41.919938 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:41.919925 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a872d741-5c2f-414a-9760-613bb4809e14" containerName="authorino" Jul 14 18:31:41.919991 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:41.919941 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a872d741-5c2f-414a-9760-613bb4809e14" containerName="authorino" Jul 14 18:31:41.920033 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:41.920027 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a872d741-5c2f-414a-9760-613bb4809e14" containerName="authorino" Jul 14 18:31:42.029062 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:42.029031 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:42.170785 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:42.170703 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:42.173297 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:42.173265 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cb7ccd7b5-hqbgr"] Jul 14 18:31:42.173378 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:42.173372 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" Jul 14 18:31:42.186185 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:42.186161 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ws7xn\"" Jul 14 18:31:42.186337 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:42.186155 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 14 18:31:42.186875 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:42.186856 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 14 18:31:42.244621 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:42.244592 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2pt6\" (UniqueName: \"kubernetes.io/projected/40bd7666-a37e-441d-8171-378a05f1ad4c-kube-api-access-d2pt6\") pod \"maas-controller-5cb7ccd7b5-hqbgr\" (UID: \"40bd7666-a37e-441d-8171-378a05f1ad4c\") " pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" Jul 14 18:31:42.244782 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:42.244723 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40bd7666-a37e-441d-8171-378a05f1ad4c-webhook-cert\") pod \"maas-controller-5cb7ccd7b5-hqbgr\" (UID: \"40bd7666-a37e-441d-8171-378a05f1ad4c\") " pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" Jul 14 18:31:42.345738 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:42.345708 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2pt6\" (UniqueName: \"kubernetes.io/projected/40bd7666-a37e-441d-8171-378a05f1ad4c-kube-api-access-d2pt6\") pod \"maas-controller-5cb7ccd7b5-hqbgr\" (UID: \"40bd7666-a37e-441d-8171-378a05f1ad4c\") " pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" Jul 14 18:31:42.345898 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:42.345877 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40bd7666-a37e-441d-8171-378a05f1ad4c-webhook-cert\") pod \"maas-controller-5cb7ccd7b5-hqbgr\" (UID: \"40bd7666-a37e-441d-8171-378a05f1ad4c\") " pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" Jul 14 18:31:42.346024 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:31:42.346008 2573 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 14 18:31:42.346073 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:31:42.346064 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40bd7666-a37e-441d-8171-378a05f1ad4c-webhook-cert podName:40bd7666-a37e-441d-8171-378a05f1ad4c nodeName:}" failed. No retries permitted until 2026-07-14 18:31:42.846047995 +0000 UTC m=+604.383495757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/40bd7666-a37e-441d-8171-378a05f1ad4c-webhook-cert") pod "maas-controller-5cb7ccd7b5-hqbgr" (UID: "40bd7666-a37e-441d-8171-378a05f1ad4c") : secret "maas-controller-webhook-cert" not found Jul 14 18:31:42.371630 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:42.371597 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2pt6\" (UniqueName: \"kubernetes.io/projected/40bd7666-a37e-441d-8171-378a05f1ad4c-kube-api-access-d2pt6\") pod \"maas-controller-5cb7ccd7b5-hqbgr\" (UID: \"40bd7666-a37e-441d-8171-378a05f1ad4c\") " pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" Jul 14 18:31:42.564949 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:42.564909 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:42.850661 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:42.850567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40bd7666-a37e-441d-8171-378a05f1ad4c-webhook-cert\") pod \"maas-controller-5cb7ccd7b5-hqbgr\" (UID: \"40bd7666-a37e-441d-8171-378a05f1ad4c\") " pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" Jul 14 18:31:42.852927 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:42.852903 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40bd7666-a37e-441d-8171-378a05f1ad4c-webhook-cert\") pod \"maas-controller-5cb7ccd7b5-hqbgr\" (UID: \"40bd7666-a37e-441d-8171-378a05f1ad4c\") " pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" Jul 14 18:31:43.084130 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:43.084098 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" Jul 14 18:31:43.246373 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:43.246341 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cb7ccd7b5-hqbgr"] Jul 14 18:31:43.246486 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:31:43.246391 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40bd7666_a37e_441d_8171_378a05f1ad4c.slice/crio-755af7081d2adf80c18a2d2203c9c4218ca7f2ea94e0f7ad0e18cee43c4223d7 WatchSource:0}: Error finding container 755af7081d2adf80c18a2d2203c9c4218ca7f2ea94e0f7ad0e18cee43c4223d7: Status 404 returned error can't find the container with id 755af7081d2adf80c18a2d2203c9c4218ca7f2ea94e0f7ad0e18cee43c4223d7 Jul 14 18:31:43.376631 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:43.376601 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" event={"ID":"40bd7666-a37e-441d-8171-378a05f1ad4c","Type":"ContainerStarted","Data":"755af7081d2adf80c18a2d2203c9c4218ca7f2ea94e0f7ad0e18cee43c4223d7"} Jul 14 18:31:45.796393 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:45.796308 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:46.063228 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:46.063148 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:31:46.400066 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:46.399975 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" event={"ID":"40bd7666-a37e-441d-8171-378a05f1ad4c","Type":"ContainerStarted","Data":"d258e5eb9d46cbd6469dbb93a72beaab4d7547affb5934036f6f51bc645859bb"} Jul 14 18:31:46.400222 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:46.400102 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" Jul 14 18:31:46.446573 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:46.446526 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" podStartSLOduration=3.17009512 podStartE2EDuration="5.446513022s" totalImagesPullingTime="2.276417902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:31:46.443214499 +0000 UTC m=+607.980662285" watchObservedRunningTime="2026-07-14 18:31:46.446513022 +0000 UTC m=+607.983960861" Jul 14 18:31:57.192739 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.192700 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cb7ccd7b5-hqbgr"] Jul 14 18:31:57.193227 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.193052 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" podUID="40bd7666-a37e-441d-8171-378a05f1ad4c" containerName="manager" containerID="cri-o://d258e5eb9d46cbd6469dbb93a72beaab4d7547affb5934036f6f51bc645859bb" gracePeriod=30 Jul 14 18:31:57.195041 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.195013 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" Jul 14 18:31:57.446027 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.445973 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" Jul 14 18:31:57.461637 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.461614 2573 generic.go:356] "Generic (PLEG): container finished" podID="40bd7666-a37e-441d-8171-378a05f1ad4c" containerID="d258e5eb9d46cbd6469dbb93a72beaab4d7547affb5934036f6f51bc645859bb" exitCode=0 Jul 14 18:31:57.461751 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.461650 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" Jul 14 18:31:57.461751 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.461678 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" event={"ID":"40bd7666-a37e-441d-8171-378a05f1ad4c","Type":"ContainerDied","Data":"d258e5eb9d46cbd6469dbb93a72beaab4d7547affb5934036f6f51bc645859bb"} Jul 14 18:31:57.461867 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.461748 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb7ccd7b5-hqbgr" event={"ID":"40bd7666-a37e-441d-8171-378a05f1ad4c","Type":"ContainerDied","Data":"755af7081d2adf80c18a2d2203c9c4218ca7f2ea94e0f7ad0e18cee43c4223d7"} Jul 14 18:31:57.461867 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.461785 2573 scope.go:117] "RemoveContainer" containerID="d258e5eb9d46cbd6469dbb93a72beaab4d7547affb5934036f6f51bc645859bb" Jul 14 18:31:57.471877 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.471859 2573 scope.go:117] "RemoveContainer" containerID="d258e5eb9d46cbd6469dbb93a72beaab4d7547affb5934036f6f51bc645859bb" Jul 14 18:31:57.472169 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:31:57.472143 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d258e5eb9d46cbd6469dbb93a72beaab4d7547affb5934036f6f51bc645859bb\": container with ID starting with d258e5eb9d46cbd6469dbb93a72beaab4d7547affb5934036f6f51bc645859bb not found: ID does not exist" containerID="d258e5eb9d46cbd6469dbb93a72beaab4d7547affb5934036f6f51bc645859bb" Jul 14 18:31:57.472276 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.472175 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d258e5eb9d46cbd6469dbb93a72beaab4d7547affb5934036f6f51bc645859bb"} err="failed to get container status \"d258e5eb9d46cbd6469dbb93a72beaab4d7547affb5934036f6f51bc645859bb\": rpc error: code = NotFound desc = could not find container \"d258e5eb9d46cbd6469dbb93a72beaab4d7547affb5934036f6f51bc645859bb\": container with ID starting with d258e5eb9d46cbd6469dbb93a72beaab4d7547affb5934036f6f51bc645859bb not found: ID does not exist" Jul 14 18:31:57.589602 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.589577 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2pt6\" (UniqueName: \"kubernetes.io/projected/40bd7666-a37e-441d-8171-378a05f1ad4c-kube-api-access-d2pt6\") pod \"40bd7666-a37e-441d-8171-378a05f1ad4c\" (UID: \"40bd7666-a37e-441d-8171-378a05f1ad4c\") " Jul 14 18:31:57.589786 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.589665 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40bd7666-a37e-441d-8171-378a05f1ad4c-webhook-cert\") pod \"40bd7666-a37e-441d-8171-378a05f1ad4c\" (UID: \"40bd7666-a37e-441d-8171-378a05f1ad4c\") " Jul 14 18:31:57.591801 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.591768 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40bd7666-a37e-441d-8171-378a05f1ad4c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "40bd7666-a37e-441d-8171-378a05f1ad4c" (UID: "40bd7666-a37e-441d-8171-378a05f1ad4c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:31:57.591894 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.591800 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bd7666-a37e-441d-8171-378a05f1ad4c-kube-api-access-d2pt6" (OuterVolumeSpecName: "kube-api-access-d2pt6") pod "40bd7666-a37e-441d-8171-378a05f1ad4c" (UID: "40bd7666-a37e-441d-8171-378a05f1ad4c"). InnerVolumeSpecName "kube-api-access-d2pt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:31:57.691088 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.691052 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2pt6\" (UniqueName: \"kubernetes.io/projected/40bd7666-a37e-441d-8171-378a05f1ad4c-kube-api-access-d2pt6\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:31:57.691088 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.691087 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40bd7666-a37e-441d-8171-378a05f1ad4c-webhook-cert\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:31:57.786349 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.786325 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cb7ccd7b5-hqbgr"] Jul 14 18:31:57.794947 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:57.794915 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cb7ccd7b5-hqbgr"] Jul 14 18:31:59.221870 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:31:59.221833 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40bd7666-a37e-441d-8171-378a05f1ad4c" path="/var/lib/kubelet/pods/40bd7666-a37e-441d-8171-378a05f1ad4c/volumes" Jul 14 18:32:08.932576 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:08.932541 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:32:17.702954 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:17.702918 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:32:17.809068 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:17.809030 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:32:18.082855 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.082775 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:32:18.134248 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.134216 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx"] Jul 14 18:32:18.134710 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.134695 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40bd7666-a37e-441d-8171-378a05f1ad4c" containerName="manager" Jul 14 18:32:18.134755 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.134712 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="40bd7666-a37e-441d-8171-378a05f1ad4c" containerName="manager" Jul 14 18:32:18.134801 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.134792 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="40bd7666-a37e-441d-8171-378a05f1ad4c" containerName="manager" Jul 14 18:32:18.138048 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.138028 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" Jul 14 18:32:18.141699 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.141681 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d45z6\"" Jul 14 18:32:18.141802 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.141685 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 14 18:32:18.155849 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.155823 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx"] Jul 14 18:32:18.178928 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.178894 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jxz\" (UniqueName: \"kubernetes.io/projected/87991a28-bafc-431a-9242-1982dc7874ab-kube-api-access-t4jxz\") pod \"maas-api-587fdf675b-jg8sx\" (UID: \"87991a28-bafc-431a-9242-1982dc7874ab\") " pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" Jul 14 18:32:18.179123 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.178959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87991a28-bafc-431a-9242-1982dc7874ab-openshift-service-ca\") pod \"maas-api-587fdf675b-jg8sx\" (UID: \"87991a28-bafc-431a-9242-1982dc7874ab\") " pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" Jul 14 18:32:18.179123 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.179015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87991a28-bafc-431a-9242-1982dc7874ab-maas-api-tls\") pod \"maas-api-587fdf675b-jg8sx\" (UID: \"87991a28-bafc-431a-9242-1982dc7874ab\") " pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" Jul 14 18:32:18.279494 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.279458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87991a28-bafc-431a-9242-1982dc7874ab-maas-api-tls\") pod \"maas-api-587fdf675b-jg8sx\" (UID: \"87991a28-bafc-431a-9242-1982dc7874ab\") " pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" Jul 14 18:32:18.279655 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.279555 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jxz\" (UniqueName: \"kubernetes.io/projected/87991a28-bafc-431a-9242-1982dc7874ab-kube-api-access-t4jxz\") pod \"maas-api-587fdf675b-jg8sx\" (UID: \"87991a28-bafc-431a-9242-1982dc7874ab\") " pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" Jul 14 18:32:18.279655 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.279588 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87991a28-bafc-431a-9242-1982dc7874ab-openshift-service-ca\") pod \"maas-api-587fdf675b-jg8sx\" (UID: \"87991a28-bafc-431a-9242-1982dc7874ab\") " pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" Jul 14 18:32:18.279655 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:32:18.279598 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 14 18:32:18.279781 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:32:18.279668 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87991a28-bafc-431a-9242-1982dc7874ab-maas-api-tls podName:87991a28-bafc-431a-9242-1982dc7874ab nodeName:}" failed. No retries permitted until 2026-07-14 18:32:18.779651035 +0000 UTC m=+640.317098799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/87991a28-bafc-431a-9242-1982dc7874ab-maas-api-tls") pod "maas-api-587fdf675b-jg8sx" (UID: "87991a28-bafc-431a-9242-1982dc7874ab") : secret "maas-api-serving-cert" not found Jul 14 18:32:18.280214 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.280193 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87991a28-bafc-431a-9242-1982dc7874ab-openshift-service-ca\") pod \"maas-api-587fdf675b-jg8sx\" (UID: \"87991a28-bafc-431a-9242-1982dc7874ab\") " pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" Jul 14 18:32:18.292917 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.292887 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jxz\" (UniqueName: \"kubernetes.io/projected/87991a28-bafc-431a-9242-1982dc7874ab-kube-api-access-t4jxz\") pod \"maas-api-587fdf675b-jg8sx\" (UID: \"87991a28-bafc-431a-9242-1982dc7874ab\") " pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" Jul 14 18:32:18.605301 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.605256 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:32:18.748149 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.748071 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:32:18.783905 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.783869 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87991a28-bafc-431a-9242-1982dc7874ab-maas-api-tls\") pod \"maas-api-587fdf675b-jg8sx\" (UID: \"87991a28-bafc-431a-9242-1982dc7874ab\") " pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" Jul 14 18:32:18.786542 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.786512 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87991a28-bafc-431a-9242-1982dc7874ab-maas-api-tls\") pod \"maas-api-587fdf675b-jg8sx\" (UID: \"87991a28-bafc-431a-9242-1982dc7874ab\") " pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" Jul 14 18:32:18.984176 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:18.984125 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:32:19.038049 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.037973 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-k8tcp"] Jul 14 18:32:19.044271 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.044250 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-k8tcp" Jul 14 18:32:19.047950 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.047929 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-x52zq\"" Jul 14 18:32:19.048049 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.047930 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 14 18:32:19.048049 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.048012 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" Jul 14 18:32:19.056501 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.056478 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-k8tcp"] Jul 14 18:32:19.087805 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.087767 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrv5x\" (UniqueName: \"kubernetes.io/projected/64f2c200-c428-4c62-ad4c-d8f13482a641-kube-api-access-jrv5x\") pod \"payload-processing-f9b8d9b9b-k8tcp\" (UID: \"64f2c200-c428-4c62-ad4c-d8f13482a641\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-k8tcp" Jul 14 18:32:19.087985 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.087829 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f2c200-c428-4c62-ad4c-d8f13482a641-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-k8tcp\" (UID: \"64f2c200-c428-4c62-ad4c-d8f13482a641\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-k8tcp" Jul 14 18:32:19.189483 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.189449 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrv5x\" (UniqueName: \"kubernetes.io/projected/64f2c200-c428-4c62-ad4c-d8f13482a641-kube-api-access-jrv5x\") pod \"payload-processing-f9b8d9b9b-k8tcp\" (UID: \"64f2c200-c428-4c62-ad4c-d8f13482a641\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-k8tcp" Jul 14 18:32:19.189680 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.189507 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f2c200-c428-4c62-ad4c-d8f13482a641-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-k8tcp\" (UID: \"64f2c200-c428-4c62-ad4c-d8f13482a641\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-k8tcp" Jul 14 18:32:19.190444 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.190417 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f2c200-c428-4c62-ad4c-d8f13482a641-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-k8tcp\" (UID: \"64f2c200-c428-4c62-ad4c-d8f13482a641\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-k8tcp" Jul 14 18:32:19.203182 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.203160 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrv5x\" (UniqueName: \"kubernetes.io/projected/64f2c200-c428-4c62-ad4c-d8f13482a641-kube-api-access-jrv5x\") pod \"payload-processing-f9b8d9b9b-k8tcp\" (UID: \"64f2c200-c428-4c62-ad4c-d8f13482a641\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-k8tcp" Jul 14 18:32:19.217465 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:32:19.217431 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87991a28_bafc_431a_9242_1982dc7874ab.slice/crio-15c404ee3ba2c0f82ca8b56d1bb87e93fa5b20de661147b6f9966644e64d3be3 WatchSource:0}: Error finding container 15c404ee3ba2c0f82ca8b56d1bb87e93fa5b20de661147b6f9966644e64d3be3: Status 404 returned error can't find the container with id 15c404ee3ba2c0f82ca8b56d1bb87e93fa5b20de661147b6f9966644e64d3be3 Jul 14 18:32:19.218842 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.218826 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:32:19.225158 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.225128 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx"] Jul 14 18:32:19.357508 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.357417 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-k8tcp" Jul 14 18:32:19.501377 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.501345 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-k8tcp"] Jul 14 18:32:19.504859 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:32:19.504827 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f2c200_c428_4c62_ad4c_d8f13482a641.slice/crio-52d48bab2c0cb67c62b41cf92eec8df3b0eece2895979cae8d83e1c40963be46 WatchSource:0}: Error finding container 52d48bab2c0cb67c62b41cf92eec8df3b0eece2895979cae8d83e1c40963be46: Status 404 returned error can't find the container with id 52d48bab2c0cb67c62b41cf92eec8df3b0eece2895979cae8d83e1c40963be46 Jul 14 18:32:19.594419 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.594377 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-k8tcp" event={"ID":"64f2c200-c428-4c62-ad4c-d8f13482a641","Type":"ContainerStarted","Data":"52d48bab2c0cb67c62b41cf92eec8df3b0eece2895979cae8d83e1c40963be46"} Jul 14 18:32:19.595598 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:19.595565 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" event={"ID":"87991a28-bafc-431a-9242-1982dc7874ab","Type":"ContainerStarted","Data":"15c404ee3ba2c0f82ca8b56d1bb87e93fa5b20de661147b6f9966644e64d3be3"} Jul 14 18:32:22.639694 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:22.639654 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-k8tcp" event={"ID":"64f2c200-c428-4c62-ad4c-d8f13482a641","Type":"ContainerStarted","Data":"baa561bdf3b1e10824cc3bf6296ee6180816e09cce4175ecf9db61bb07630c5d"} Jul 14 18:32:22.640172 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:22.639784 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-k8tcp" Jul 14 18:32:22.641317 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:22.641271 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" event={"ID":"87991a28-bafc-431a-9242-1982dc7874ab","Type":"ContainerStarted","Data":"e05445f7721ebbcdc67af2afbcd1f7c17e4f548dd176bfdc2ca1071cf612f643"} Jul 14 18:32:22.641447 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:22.641340 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" Jul 14 18:32:22.667997 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:22.667951 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-k8tcp" podStartSLOduration=1.387950491 podStartE2EDuration="3.66793748s" totalImagesPullingTime="2.279986989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:32:22.666376985 +0000 UTC m=+644.203824771" watchObservedRunningTime="2026-07-14 18:32:22.66793748 +0000 UTC m=+644.205385266" Jul 14 18:32:22.693359 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:22.693307 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" podStartSLOduration=2.1274744070000002 podStartE2EDuration="4.693272518s" totalImagesPullingTime="2.565798111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:32:22.692901973 +0000 UTC m=+644.230349760" watchObservedRunningTime="2026-07-14 18:32:22.693272518 +0000 UTC m=+644.230720304" Jul 14 18:32:24.920353 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:24.920311 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:32:25.019651 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:25.019611 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:32:27.671468 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:27.671435 2573 generic.go:356] "Generic (PLEG): container finished" podID="83981a26-6460-417f-9442-602373954802" containerID="7d6ad8f02e97a8d80c6f30dd5cb1ef2d80ede30029d8dde19696a943bc268031" exitCode=0 Jul 14 18:32:27.671890 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:27.671509 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" event={"ID":"83981a26-6460-417f-9442-602373954802","Type":"ContainerDied","Data":"7d6ad8f02e97a8d80c6f30dd5cb1ef2d80ede30029d8dde19696a943bc268031"} Jul 14 18:32:27.671933 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:27.671896 2573 scope.go:117] "RemoveContainer" containerID="7d6ad8f02e97a8d80c6f30dd5cb1ef2d80ede30029d8dde19696a943bc268031" Jul 14 18:32:27.672951 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:27.672926 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:32:27.673010 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:27.672991 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:32:27.673047 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:27.673027 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 14 18:32:28.653314 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:28.653271 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-587fdf675b-jg8sx" Jul 14 18:32:28.680231 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:28.680199 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" event={"ID":"83981a26-6460-417f-9442-602373954802","Type":"ContainerStarted","Data":"2a672b2b0163d3945649c83b71263b64ef62acb1115c8a957938f00602888ae4"} Jul 14 18:32:28.842214 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:28.842179 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:32:28.843448 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:28.843425 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rlbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 18:32:28.843528 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:28.843476 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" podUID="83981a26-6460-417f-9442-602373954802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 18:32:29.701300 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:29.701262 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:32:29.809066 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:29.808955 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:32:29.842468 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:29.842438 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rlbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 18:32:29.842617 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:29.842486 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" podUID="83981a26-6460-417f-9442-602373954802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 18:32:30.842671 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:30.842638 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rlbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 18:32:30.843066 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:30.842693 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" podUID="83981a26-6460-417f-9442-602373954802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 18:32:31.848743 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:31.848702 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rlbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 14 18:32:31.849120 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:31.848875 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" podUID="83981a26-6460-417f-9442-602373954802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 14 18:32:32.577788 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:32.577754 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:32:32.845535 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:32.845455 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:32:32.845779 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:32.845758 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:32:32.846489 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:32.846465 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rlbm8" Jul 14 18:32:33.649091 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:33.649067 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-k8tcp" Jul 14 18:32:45.509507 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:45.509466 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:32:50.486919 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.486882 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z"] Jul 14 18:32:50.492186 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.492167 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.496549 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.496429 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 18:32:50.496549 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.496501 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 18:32:50.497208 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.496720 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bqqxv\"" Jul 14 18:32:50.497208 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.496763 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 14 18:32:50.503310 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.502678 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z"] Jul 14 18:32:50.519980 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.519952 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e80dcfd-345e-49fb-be55-a3f8770a9903-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.520134 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.520006 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e80dcfd-345e-49fb-be55-a3f8770a9903-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.520134 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.520098 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e80dcfd-345e-49fb-be55-a3f8770a9903-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.520264 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.520139 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e80dcfd-345e-49fb-be55-a3f8770a9903-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.520264 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.520181 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e80dcfd-345e-49fb-be55-a3f8770a9903-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.520264 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.520243 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmq9h\" (UniqueName: \"kubernetes.io/projected/0e80dcfd-345e-49fb-be55-a3f8770a9903-kube-api-access-fmq9h\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.576261 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.576224 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:32:50.621326 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.621268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e80dcfd-345e-49fb-be55-a3f8770a9903-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.621475 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.621336 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e80dcfd-345e-49fb-be55-a3f8770a9903-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.621475 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.621376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmq9h\" (UniqueName: \"kubernetes.io/projected/0e80dcfd-345e-49fb-be55-a3f8770a9903-kube-api-access-fmq9h\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.621475 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.621405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e80dcfd-345e-49fb-be55-a3f8770a9903-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.621475 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.621428 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e80dcfd-345e-49fb-be55-a3f8770a9903-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.621693 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.621483 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e80dcfd-345e-49fb-be55-a3f8770a9903-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.621754 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.621728 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e80dcfd-345e-49fb-be55-a3f8770a9903-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.621799 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.621754 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e80dcfd-345e-49fb-be55-a3f8770a9903-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.621799 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.621787 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e80dcfd-345e-49fb-be55-a3f8770a9903-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.623659 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.623632 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e80dcfd-345e-49fb-be55-a3f8770a9903-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.623927 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.623908 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e80dcfd-345e-49fb-be55-a3f8770a9903-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.631142 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.631120 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmq9h\" (UniqueName: \"kubernetes.io/projected/0e80dcfd-345e-49fb-be55-a3f8770a9903-kube-api-access-fmq9h\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gpk6z\" (UID: \"0e80dcfd-345e-49fb-be55-a3f8770a9903\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.805338 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.805226 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:32:50.976714 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:50.976690 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z"] Jul 14 18:32:50.979340 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:32:50.979309 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e80dcfd_345e_49fb_be55_a3f8770a9903.slice/crio-be1c726e9a562f6b0aff26fad3749a3c414129d7b09eca0c84a9613dcb0ef41e WatchSource:0}: Error finding container be1c726e9a562f6b0aff26fad3749a3c414129d7b09eca0c84a9613dcb0ef41e: Status 404 returned error can't find the container with id be1c726e9a562f6b0aff26fad3749a3c414129d7b09eca0c84a9613dcb0ef41e Jul 14 18:32:51.809300 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:51.809255 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" event={"ID":"0e80dcfd-345e-49fb-be55-a3f8770a9903","Type":"ContainerStarted","Data":"be1c726e9a562f6b0aff26fad3749a3c414129d7b09eca0c84a9613dcb0ef41e"} Jul 14 18:32:56.841099 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:56.841060 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" event={"ID":"0e80dcfd-345e-49fb-be55-a3f8770a9903","Type":"ContainerStarted","Data":"45408d749197ce9a85d576ccf574bae566434ee15f6ac4afdce8311dc89fedce"} Jul 14 18:32:56.866624 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:56.866575 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" podStartSLOduration=1.413361676 podStartE2EDuration="6.866560192s" totalImagesPullingTime="5.453198516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:32:56.864369919 +0000 UTC m=+678.401817703" watchObservedRunningTime="2026-07-14 18:32:56.866560192 +0000 UTC m=+678.404008023" Jul 14 18:32:58.202010 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.201978 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-j869s"] Jul 14 18:32:58.206107 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.206087 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.209832 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.209812 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 14 18:32:58.220444 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.220421 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-j869s"] Jul 14 18:32:58.222791 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.222752 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:32:58.302650 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.302615 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f757be3-78b0-4951-9147-6f8102a02cea-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.302813 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.302668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f757be3-78b0-4951-9147-6f8102a02cea-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.302813 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.302808 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f757be3-78b0-4951-9147-6f8102a02cea-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.302928 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.302855 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ffn\" (UniqueName: \"kubernetes.io/projected/5f757be3-78b0-4951-9147-6f8102a02cea-kube-api-access-m7ffn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.303057 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.303022 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f757be3-78b0-4951-9147-6f8102a02cea-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.303155 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.303137 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f757be3-78b0-4951-9147-6f8102a02cea-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.404232 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.404197 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f757be3-78b0-4951-9147-6f8102a02cea-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.404232 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.404238 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f757be3-78b0-4951-9147-6f8102a02cea-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.404488 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.404255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f757be3-78b0-4951-9147-6f8102a02cea-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.404488 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.404337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f757be3-78b0-4951-9147-6f8102a02cea-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.404488 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.404361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ffn\" (UniqueName: \"kubernetes.io/projected/5f757be3-78b0-4951-9147-6f8102a02cea-kube-api-access-m7ffn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.404488 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.404405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f757be3-78b0-4951-9147-6f8102a02cea-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.404695 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.404616 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f757be3-78b0-4951-9147-6f8102a02cea-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.404756 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.404737 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f757be3-78b0-4951-9147-6f8102a02cea-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.404847 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.404826 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f757be3-78b0-4951-9147-6f8102a02cea-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.406582 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.406558 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f757be3-78b0-4951-9147-6f8102a02cea-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.406701 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.406682 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f757be3-78b0-4951-9147-6f8102a02cea-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.418039 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.418013 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ffn\" (UniqueName: \"kubernetes.io/projected/5f757be3-78b0-4951-9147-6f8102a02cea-kube-api-access-m7ffn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-j869s\" (UID: \"5f757be3-78b0-4951-9147-6f8102a02cea\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.518070 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.517990 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:32:58.663639 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.663614 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-j869s"] Jul 14 18:32:58.665272 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:32:58.665248 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f757be3_78b0_4951_9147_6f8102a02cea.slice/crio-487d860da1f91cbcc6306d353a8247762dd7e8c5fe062a587f30c905f0879f98 WatchSource:0}: Error finding container 487d860da1f91cbcc6306d353a8247762dd7e8c5fe062a587f30c905f0879f98: Status 404 returned error can't find the container with id 487d860da1f91cbcc6306d353a8247762dd7e8c5fe062a587f30c905f0879f98 Jul 14 18:32:58.854092 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:58.854019 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" event={"ID":"5f757be3-78b0-4951-9147-6f8102a02cea","Type":"ContainerStarted","Data":"487d860da1f91cbcc6306d353a8247762dd7e8c5fe062a587f30c905f0879f98"} Jul 14 18:32:59.861520 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:59.861480 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" event={"ID":"5f757be3-78b0-4951-9147-6f8102a02cea","Type":"ContainerStarted","Data":"4f38d119aa7020f71fefb507f9c98ccf1c0367a4f11ae87b34f0bacdd4ed4153"} Jul 14 18:32:59.885184 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:32:59.885121 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" podStartSLOduration=1.6894986539999999 podStartE2EDuration="1.885097149s" totalImagesPullingTime="195.598495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:32:59.882643736 +0000 UTC m=+681.420091521" watchObservedRunningTime="2026-07-14 18:32:59.885097149 +0000 UTC m=+681.422544938" Jul 14 18:33:00.806365 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:33:00.806323 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:33:00.806539 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:33:00.806394 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:33:00.818943 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:33:00.818918 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:33:00.878862 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:33:00.878836 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gpk6z" Jul 14 18:33:01.739190 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:33:01.739155 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:33:08.518841 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:33:08.518809 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:33:08.519217 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:33:08.518867 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:33:08.531370 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:33:08.531350 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:33:08.923314 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:33:08.923218 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-j869s" Jul 14 18:33:17.153696 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:33:17.153665 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:33:24.015496 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:33:24.015455 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:33:30.177656 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:33:30.177623 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:33:30.292883 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:33:30.292851 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:34:04.692710 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:04.692618 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:34:04.792336 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:04.792272 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:34:29.853570 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:29.853527 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55dcc8665-gw4fn"] Jul 14 18:34:29.854033 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:29.853747 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-55dcc8665-gw4fn" podUID="a033555e-12ba-4a20-9358-0e122a1d0cfc" containerName="authorino" containerID="cri-o://185dc92d430f3529fd731de2ff17b120666151ba30cc6fbdb78c578bb025c91e" gracePeriod=30 Jul 14 18:34:30.118709 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.118651 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55dcc8665-gw4fn" Jul 14 18:34:30.199552 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.199525 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l668t\" (UniqueName: \"kubernetes.io/projected/a033555e-12ba-4a20-9358-0e122a1d0cfc-kube-api-access-l668t\") pod \"a033555e-12ba-4a20-9358-0e122a1d0cfc\" (UID: \"a033555e-12ba-4a20-9358-0e122a1d0cfc\") " Jul 14 18:34:30.199702 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.199568 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a033555e-12ba-4a20-9358-0e122a1d0cfc-tls-cert\") pod \"a033555e-12ba-4a20-9358-0e122a1d0cfc\" (UID: \"a033555e-12ba-4a20-9358-0e122a1d0cfc\") " Jul 14 18:34:30.201551 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.201524 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a033555e-12ba-4a20-9358-0e122a1d0cfc-kube-api-access-l668t" (OuterVolumeSpecName: "kube-api-access-l668t") pod "a033555e-12ba-4a20-9358-0e122a1d0cfc" (UID: "a033555e-12ba-4a20-9358-0e122a1d0cfc"). InnerVolumeSpecName "kube-api-access-l668t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:34:30.210981 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.210960 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a033555e-12ba-4a20-9358-0e122a1d0cfc-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a033555e-12ba-4a20-9358-0e122a1d0cfc" (UID: "a033555e-12ba-4a20-9358-0e122a1d0cfc"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:34:30.301264 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.301238 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l668t\" (UniqueName: \"kubernetes.io/projected/a033555e-12ba-4a20-9358-0e122a1d0cfc-kube-api-access-l668t\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:34:30.301264 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.301262 2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a033555e-12ba-4a20-9358-0e122a1d0cfc-tls-cert\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:34:30.377755 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.377694 2573 generic.go:356] "Generic (PLEG): container finished" podID="a033555e-12ba-4a20-9358-0e122a1d0cfc" containerID="185dc92d430f3529fd731de2ff17b120666151ba30cc6fbdb78c578bb025c91e" exitCode=0 Jul 14 18:34:30.377755 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.377741 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55dcc8665-gw4fn" Jul 14 18:34:30.377885 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.377770 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55dcc8665-gw4fn" event={"ID":"a033555e-12ba-4a20-9358-0e122a1d0cfc","Type":"ContainerDied","Data":"185dc92d430f3529fd731de2ff17b120666151ba30cc6fbdb78c578bb025c91e"} Jul 14 18:34:30.377885 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.377843 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55dcc8665-gw4fn" event={"ID":"a033555e-12ba-4a20-9358-0e122a1d0cfc","Type":"ContainerDied","Data":"f2b3050ec219ae5c3647933c6958bfa823bad9c6ceaf3d142fac6351628e7faa"} Jul 14 18:34:30.377885 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.377871 2573 scope.go:117] "RemoveContainer" containerID="185dc92d430f3529fd731de2ff17b120666151ba30cc6fbdb78c578bb025c91e" Jul 14 18:34:30.387980 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.387961 2573 scope.go:117] "RemoveContainer" containerID="185dc92d430f3529fd731de2ff17b120666151ba30cc6fbdb78c578bb025c91e" Jul 14 18:34:30.388237 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:34:30.388215 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"185dc92d430f3529fd731de2ff17b120666151ba30cc6fbdb78c578bb025c91e\": container with ID starting with 185dc92d430f3529fd731de2ff17b120666151ba30cc6fbdb78c578bb025c91e not found: ID does not exist" containerID="185dc92d430f3529fd731de2ff17b120666151ba30cc6fbdb78c578bb025c91e" Jul 14 18:34:30.388324 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.388244 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"185dc92d430f3529fd731de2ff17b120666151ba30cc6fbdb78c578bb025c91e"} err="failed to get container status \"185dc92d430f3529fd731de2ff17b120666151ba30cc6fbdb78c578bb025c91e\": rpc error: code = NotFound desc = could not find container \"185dc92d430f3529fd731de2ff17b120666151ba30cc6fbdb78c578bb025c91e\": container with ID starting with 185dc92d430f3529fd731de2ff17b120666151ba30cc6fbdb78c578bb025c91e not found: ID does not exist" Jul 14 18:34:30.410798 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.410771 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55dcc8665-gw4fn"] Jul 14 18:34:30.415590 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.415562 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-55dcc8665-gw4fn"] Jul 14 18:34:30.423559 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.423534 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:34:30.518767 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:30.518742 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:34:31.222773 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:31.222741 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a033555e-12ba-4a20-9358-0e122a1d0cfc" path="/var/lib/kubelet/pods/a033555e-12ba-4a20-9358-0e122a1d0cfc/volumes" Jul 14 18:34:31.710371 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:31.710337 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:34:31.803240 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:31.803206 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:34:38.896001 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:38.895969 2573 scope.go:117] "RemoveContainer" containerID="efa60b16ab4acf3bfce25fa31e5f16b520be1ae765fdab3051d3cbc66151ec35" Jul 14 18:34:55.647048 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:34:55.647005 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:06.140418 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:06.140385 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:06.241071 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:06.241035 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:06.304025 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:06.303996 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:11.291626 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:11.291590 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:11.391589 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:11.391555 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:44.575460 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:44.575422 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:45.829547 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:45.829517 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:35:45.956459 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:45.956429 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-846475b4dc-cj4kt"] Jul 14 18:35:45.956962 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:45.956948 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a033555e-12ba-4a20-9358-0e122a1d0cfc" containerName="authorino" Jul 14 18:35:45.957006 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:45.956966 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a033555e-12ba-4a20-9358-0e122a1d0cfc" containerName="authorino" Jul 14 18:35:45.957057 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:45.957048 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a033555e-12ba-4a20-9358-0e122a1d0cfc" containerName="authorino" Jul 14 18:35:45.960420 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:45.960400 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-846475b4dc-cj4kt" Jul 14 18:35:45.964602 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:45.964581 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ws7xn\"" Jul 14 18:35:45.964740 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:45.964640 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 14 18:35:45.964740 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:45.964581 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 14 18:35:45.969110 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:45.969087 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-846475b4dc-cj4kt"] Jul 14 18:35:46.026391 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:46.026366 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ddf090f3-ed96-418b-85ef-9f6f25f387e6-webhook-cert\") pod \"maas-controller-846475b4dc-cj4kt\" (UID: \"ddf090f3-ed96-418b-85ef-9f6f25f387e6\") " pod="opendatahub/maas-controller-846475b4dc-cj4kt" Jul 14 18:35:46.026538 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:46.026407 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rr5t\" (UniqueName: \"kubernetes.io/projected/ddf090f3-ed96-418b-85ef-9f6f25f387e6-kube-api-access-4rr5t\") pod \"maas-controller-846475b4dc-cj4kt\" (UID: \"ddf090f3-ed96-418b-85ef-9f6f25f387e6\") " pod="opendatahub/maas-controller-846475b4dc-cj4kt" Jul 14 18:35:46.127404 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:46.127337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rr5t\" (UniqueName: \"kubernetes.io/projected/ddf090f3-ed96-418b-85ef-9f6f25f387e6-kube-api-access-4rr5t\") pod \"maas-controller-846475b4dc-cj4kt\" (UID: \"ddf090f3-ed96-418b-85ef-9f6f25f387e6\") " pod="opendatahub/maas-controller-846475b4dc-cj4kt" Jul 14 18:35:46.127522 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:46.127456 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ddf090f3-ed96-418b-85ef-9f6f25f387e6-webhook-cert\") pod \"maas-controller-846475b4dc-cj4kt\" (UID: \"ddf090f3-ed96-418b-85ef-9f6f25f387e6\") " pod="opendatahub/maas-controller-846475b4dc-cj4kt" Jul 14 18:35:46.129837 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:46.129817 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ddf090f3-ed96-418b-85ef-9f6f25f387e6-webhook-cert\") pod \"maas-controller-846475b4dc-cj4kt\" (UID: \"ddf090f3-ed96-418b-85ef-9f6f25f387e6\") " pod="opendatahub/maas-controller-846475b4dc-cj4kt" Jul 14 18:35:46.137356 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:46.137336 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rr5t\" (UniqueName: \"kubernetes.io/projected/ddf090f3-ed96-418b-85ef-9f6f25f387e6-kube-api-access-4rr5t\") pod \"maas-controller-846475b4dc-cj4kt\" (UID: \"ddf090f3-ed96-418b-85ef-9f6f25f387e6\") " pod="opendatahub/maas-controller-846475b4dc-cj4kt" Jul 14 18:35:46.272350 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:46.272321 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-846475b4dc-cj4kt" Jul 14 18:35:46.412719 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:46.412690 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-846475b4dc-cj4kt"] Jul 14 18:35:46.414296 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:35:46.414258 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf090f3_ed96_418b_85ef_9f6f25f387e6.slice/crio-56f6af41210a5c50404831937b6302bcfa588678be914ae05d9f6ea316664c07 WatchSource:0}: Error finding container 56f6af41210a5c50404831937b6302bcfa588678be914ae05d9f6ea316664c07: Status 404 returned error can't find the container with id 56f6af41210a5c50404831937b6302bcfa588678be914ae05d9f6ea316664c07 Jul 14 18:35:46.807835 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:46.807801 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-846475b4dc-cj4kt" event={"ID":"ddf090f3-ed96-418b-85ef-9f6f25f387e6","Type":"ContainerStarted","Data":"56f6af41210a5c50404831937b6302bcfa588678be914ae05d9f6ea316664c07"} Jul 14 18:35:47.814505 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:47.814472 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-846475b4dc-cj4kt" event={"ID":"ddf090f3-ed96-418b-85ef-9f6f25f387e6","Type":"ContainerStarted","Data":"25e7c6272725bf59765785bf6095c1674f2d99a79ae18c7e9bb7971a6c94bef7"} Jul 14 18:35:47.814947 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:47.814530 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-846475b4dc-cj4kt" Jul 14 18:35:47.837491 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:47.837441 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-846475b4dc-cj4kt" podStartSLOduration=2.366572679 podStartE2EDuration="2.837427018s" totalImagesPullingTime="470.854339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:35:47.834818296 +0000 UTC m=+849.372266082" watchObservedRunningTime="2026-07-14 18:35:47.837427018 +0000 UTC m=+849.374874803" Jul 14 18:35:58.823116 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:35:58.823087 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-846475b4dc-cj4kt" Jul 14 18:36:28.755200 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:36:28.755130 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:28.857571 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:36:28.857507 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:34.099564 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:36:34.099476 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:34.242682 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:36:34.242651 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:34.338422 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:36:34.338393 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:34.493703 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:36:34.493673 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:38.883817 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:36:38.883788 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:36:38.885460 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:36:38.885439 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:36:38.890941 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:36:38.890918 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:36:38.892769 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:36:38.892753 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:36:40.715890 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:36:40.715857 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:36:40.785856 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:36:40.785826 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:38:11.514877 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:38:11.514844 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:40:22.587001 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:40:22.586968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:29.580230 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:41:29.580196 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:41:38.928717 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:41:38.928685 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:41:38.929382 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:41:38.929363 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:41:38.936029 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:41:38.936005 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:41:38.936784 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:41:38.936766 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:42:04.009950 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:42:04.009916 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:45:00.148720 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:00.148680 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734245-r5vrd"] Jul 14 18:45:00.152624 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:00.152606 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-r5vrd" Jul 14 18:45:00.155620 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:00.155598 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 18:45:00.162444 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:00.162427 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-r5vrd" Jul 14 18:45:00.168637 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:00.168582 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734245-r5vrd"] Jul 14 18:45:00.516906 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:00.516883 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734245-r5vrd"] Jul 14 18:45:00.519306 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:45:00.519267 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1155c96_7e2e_4e90_9222_5cc5517a4708.slice/crio-5f2c7c99bff15e3453603ccebd075ea04ac88831edbfdf10a4f7441cffaebee7 WatchSource:0}: Error finding container 5f2c7c99bff15e3453603ccebd075ea04ac88831edbfdf10a4f7441cffaebee7: Status 404 returned error can't find the container with id 5f2c7c99bff15e3453603ccebd075ea04ac88831edbfdf10a4f7441cffaebee7 Jul 14 18:45:00.521228 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:00.521210 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:45:00.936807 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:00.936715 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-r5vrd" event={"ID":"c1155c96-7e2e-4e90-9222-5cc5517a4708","Type":"ContainerStarted","Data":"5f2c7c99bff15e3453603ccebd075ea04ac88831edbfdf10a4f7441cffaebee7"} Jul 14 18:45:02.482654 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:02.482617 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:45:02.949798 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:02.949764 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-r5vrd" event={"ID":"c1155c96-7e2e-4e90-9222-5cc5517a4708","Type":"ContainerStarted","Data":"d3499a582912250dfe6809da0ac099091979700768e92b2cc767026ed174977a"} Jul 14 18:45:02.970135 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:02.970090 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-r5vrd" podStartSLOduration=0.827691437 podStartE2EDuration="2.970075215s" totalImagesPullingTime="2.142383778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:45:02.968349351 +0000 UTC m=+1404.505797137" watchObservedRunningTime="2026-07-14 18:45:02.970075215 +0000 UTC m=+1404.507523000" Jul 14 18:45:03.956672 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:03.956640 2573 generic.go:356] "Generic (PLEG): container finished" podID="c1155c96-7e2e-4e90-9222-5cc5517a4708" containerID="d3499a582912250dfe6809da0ac099091979700768e92b2cc767026ed174977a" exitCode=0 Jul 14 18:45:03.957056 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:03.956730 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-r5vrd" event={"ID":"c1155c96-7e2e-4e90-9222-5cc5517a4708","Type":"ContainerDied","Data":"d3499a582912250dfe6809da0ac099091979700768e92b2cc767026ed174977a"} Jul 14 18:45:05.106357 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:05.106332 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-r5vrd" Jul 14 18:45:05.970404 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:05.970374 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-r5vrd" Jul 14 18:45:05.970607 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:05.970406 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734245-r5vrd" event={"ID":"c1155c96-7e2e-4e90-9222-5cc5517a4708","Type":"ContainerDied","Data":"5f2c7c99bff15e3453603ccebd075ea04ac88831edbfdf10a4f7441cffaebee7"} Jul 14 18:45:05.970607 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:45:05.970453 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f2c7c99bff15e3453603ccebd075ea04ac88831edbfdf10a4f7441cffaebee7" Jul 14 18:46:38.969428 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:46:38.969397 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:46:38.972071 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:46:38.972051 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:46:38.977058 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:46:38.977043 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:46:38.979688 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:46:38.979672 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:47:31.727988 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:47:31.727955 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:38.283896 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:47:38.283855 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:47.206006 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:47:47.205971 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:51.762994 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:47:51.762956 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:55.102241 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:47:55.102205 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:55.198888 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:47:55.198851 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:47:55.607985 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:47:55.607957 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:01.892678 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:01.892647 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:01.995377 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:01.995344 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:02.144393 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:02.144270 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:02.239197 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:02.239164 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:03.324074 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.324013 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:03.476520 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.476481 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:03.588972 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.588897 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf"] Jul 14 18:48:03.589464 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.589449 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1155c96-7e2e-4e90-9222-5cc5517a4708" containerName="cleanup" Jul 14 18:48:03.589522 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.589465 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1155c96-7e2e-4e90-9222-5cc5517a4708" containerName="cleanup" Jul 14 18:48:03.589559 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.589549 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1155c96-7e2e-4e90-9222-5cc5517a4708" containerName="cleanup" Jul 14 18:48:03.592816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.592801 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" Jul 14 18:48:03.595624 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.595597 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bfd487d2-serving-cert\"" Jul 14 18:48:03.604709 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.604690 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf"] Jul 14 18:48:03.623497 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.623472 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:03.674455 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.674426 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgnc\" (UniqueName: \"kubernetes.io/projected/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-kube-api-access-dtgnc\") pod \"maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf\" (UID: \"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" Jul 14 18:48:03.674572 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.674459 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-openshift-service-ca\") pod \"maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf\" (UID: \"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" Jul 14 18:48:03.674572 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.674538 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-maas-api-tls\") pod \"maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf\" (UID: \"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" Jul 14 18:48:03.775407 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.775379 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgnc\" (UniqueName: \"kubernetes.io/projected/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-kube-api-access-dtgnc\") pod \"maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf\" (UID: \"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" Jul 14 18:48:03.775564 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.775419 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-openshift-service-ca\") pod \"maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf\" (UID: \"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" Jul 14 18:48:03.775564 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.775510 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-maas-api-tls\") pod \"maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf\" (UID: \"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" Jul 14 18:48:03.776163 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.776137 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-openshift-service-ca\") pod \"maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf\" (UID: \"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" Jul 14 18:48:03.777936 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.777911 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-maas-api-tls\") pod \"maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf\" (UID: \"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" Jul 14 18:48:03.788124 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.788102 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgnc\" (UniqueName: \"kubernetes.io/projected/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-kube-api-access-dtgnc\") pod \"maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf\" (UID: \"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" Jul 14 18:48:03.828327 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.828246 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:03.903806 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:03.903747 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" Jul 14 18:48:04.056133 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:04.056093 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf"] Jul 14 18:48:04.057734 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:48:04.057705 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c2c77a_f360_4c1e_a2d7_6a01232ebe0d.slice/crio-c582e6cff427663baa41fe0d0f249e6868c467e4037c0962b318c1d128590b7d WatchSource:0}: Error finding container c582e6cff427663baa41fe0d0f249e6868c467e4037c0962b318c1d128590b7d: Status 404 returned error can't find the container with id c582e6cff427663baa41fe0d0f249e6868c467e4037c0962b318c1d128590b7d Jul 14 18:48:04.171938 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:04.171853 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:04.272485 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:04.272442 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:05.015875 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:05.015838 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" event={"ID":"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d","Type":"ContainerStarted","Data":"3ef12d1705da90dd9fd75aac285ca8494e1d55f4890d09d724e455ad7b054baa"} Jul 14 18:48:05.016308 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:05.015921 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" Jul 14 18:48:05.016308 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:05.015937 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" event={"ID":"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d","Type":"ContainerStarted","Data":"c582e6cff427663baa41fe0d0f249e6868c467e4037c0962b318c1d128590b7d"} Jul 14 18:48:05.035404 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:05.035358 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" podStartSLOduration=1.543426746 podStartE2EDuration="2.035342625s" totalImagesPullingTime="491.915879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:48:05.034672607 +0000 UTC m=+1586.572120393" watchObservedRunningTime="2026-07-14 18:48:05.035342625 +0000 UTC m=+1586.572790409" Jul 14 18:48:06.073278 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:06.073246 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:48:08.413669 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:08.413637 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l"] Jul 14 18:48:08.417401 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:08.417384 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l" Jul 14 18:48:08.420450 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:08.420425 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 18:48:08.425202 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:08.425177 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l"] Jul 14 18:48:08.523141 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:08.523108 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e79146-dc86-42e9-9627-a39d5719aab9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l\" (UID: \"92e79146-dc86-42e9-9627-a39d5719aab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l" Jul 14 18:48:08.624492 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:08.624456 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e79146-dc86-42e9-9627-a39d5719aab9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l\" (UID: \"92e79146-dc86-42e9-9627-a39d5719aab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l" Jul 14 18:48:08.625202 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:08.625181 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e79146-dc86-42e9-9627-a39d5719aab9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l\" (UID: \"92e79146-dc86-42e9-9627-a39d5719aab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l" Jul 14 18:48:08.732574 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:08.732542 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l" Jul 14 18:48:08.877616 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:08.877588 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l"] Jul 14 18:48:08.879082 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:48:08.879056 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e79146_dc86_42e9_9627_a39d5719aab9.slice/crio-8cb5ac68abefaa38f8396ce5ada48d60d1270b6995f56b92fe58e2493918871f WatchSource:0}: Error finding container 8cb5ac68abefaa38f8396ce5ada48d60d1270b6995f56b92fe58e2493918871f: Status 404 returned error can't find the container with id 8cb5ac68abefaa38f8396ce5ada48d60d1270b6995f56b92fe58e2493918871f Jul 14 18:48:09.049967 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:09.049870 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l" event={"ID":"92e79146-dc86-42e9-9627-a39d5719aab9","Type":"ContainerStarted","Data":"8cb5ac68abefaa38f8396ce5ada48d60d1270b6995f56b92fe58e2493918871f"} Jul 14 18:48:10.057495 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:10.057458 2573 generic.go:356] "Generic (PLEG): container finished" podID="92e79146-dc86-42e9-9627-a39d5719aab9" containerID="fe2c94510d3458d17f158d82b9520d709ca9d32301872bbc475a952d0586b97f" exitCode=7 Jul 14 18:48:10.057948 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:10.057541 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l" event={"ID":"92e79146-dc86-42e9-9627-a39d5719aab9","Type":"ContainerDied","Data":"fe2c94510d3458d17f158d82b9520d709ca9d32301872bbc475a952d0586b97f"} Jul 14 18:48:10.057948 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:10.057775 2573 scope.go:117] "RemoveContainer" containerID="fe2c94510d3458d17f158d82b9520d709ca9d32301872bbc475a952d0586b97f" Jul 14 18:48:11.027600 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:11.027567 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" Jul 14 18:48:11.067275 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:11.067240 2573 generic.go:356] "Generic (PLEG): container finished" podID="92e79146-dc86-42e9-9627-a39d5719aab9" containerID="7d00a57937ac69314a0d538bc66fc381588c02220faee5fbbe0a5098702242de" exitCode=7 Jul 14 18:48:11.067711 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:11.067329 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l" event={"ID":"92e79146-dc86-42e9-9627-a39d5719aab9","Type":"ContainerDied","Data":"7d00a57937ac69314a0d538bc66fc381588c02220faee5fbbe0a5098702242de"} Jul 14 18:48:11.067711 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:11.067405 2573 scope.go:117] "RemoveContainer" containerID="fe2c94510d3458d17f158d82b9520d709ca9d32301872bbc475a952d0586b97f" Jul 14 18:48:11.067711 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:11.067585 2573 scope.go:117] "RemoveContainer" containerID="7d00a57937ac69314a0d538bc66fc381588c02220faee5fbbe0a5098702242de" Jul 14 18:48:11.067948 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:48:11.067861 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l_odh-ai-gateway-infra(92e79146-dc86-42e9-9627-a39d5719aab9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l" podUID="92e79146-dc86-42e9-9627-a39d5719aab9" Jul 14 18:48:12.074235 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:12.074213 2573 scope.go:117] "RemoveContainer" containerID="7d00a57937ac69314a0d538bc66fc381588c02220faee5fbbe0a5098702242de" Jul 14 18:48:12.074562 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:48:12.074429 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l_odh-ai-gateway-infra(92e79146-dc86-42e9-9627-a39d5719aab9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l" podUID="92e79146-dc86-42e9-9627-a39d5719aab9" Jul 14 18:48:25.216035 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:25.215995 2573 scope.go:117] "RemoveContainer" containerID="7d00a57937ac69314a0d538bc66fc381588c02220faee5fbbe0a5098702242de" Jul 14 18:48:26.155890 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:26.155854 2573 generic.go:356] "Generic (PLEG): container finished" podID="92e79146-dc86-42e9-9627-a39d5719aab9" containerID="b08058ab152f2e670db2865149d17ec2e479844a48f3b8f2ec499783f22f76fe" exitCode=0 Jul 14 18:48:26.156060 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:26.155927 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l" event={"ID":"92e79146-dc86-42e9-9627-a39d5719aab9","Type":"ContainerDied","Data":"b08058ab152f2e670db2865149d17ec2e479844a48f3b8f2ec499783f22f76fe"} Jul 14 18:48:26.156060 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:26.156017 2573 scope.go:117] "RemoveContainer" containerID="7d00a57937ac69314a0d538bc66fc381588c02220faee5fbbe0a5098702242de" Jul 14 18:48:27.191135 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:27.191101 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l"] Jul 14 18:48:27.303452 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:27.303426 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l" Jul 14 18:48:27.405135 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:27.405106 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e79146-dc86-42e9-9627-a39d5719aab9-maas-api-service-ca\") pod \"92e79146-dc86-42e9-9627-a39d5719aab9\" (UID: \"92e79146-dc86-42e9-9627-a39d5719aab9\") " Jul 14 18:48:27.405475 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:27.405453 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e79146-dc86-42e9-9627-a39d5719aab9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92e79146-dc86-42e9-9627-a39d5719aab9" (UID: "92e79146-dc86-42e9-9627-a39d5719aab9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:48:27.506377 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:27.506349 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92e79146-dc86-42e9-9627-a39d5719aab9-maas-api-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:48:28.171195 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:28.171167 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l" Jul 14 18:48:28.171195 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:28.171190 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l" event={"ID":"92e79146-dc86-42e9-9627-a39d5719aab9","Type":"ContainerDied","Data":"8cb5ac68abefaa38f8396ce5ada48d60d1270b6995f56b92fe58e2493918871f"} Jul 14 18:48:28.171507 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:28.171235 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cb5ac68abefaa38f8396ce5ada48d60d1270b6995f56b92fe58e2493918871f" Jul 14 18:48:28.229674 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:28.229646 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l"] Jul 14 18:48:28.237702 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:28.237669 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-94z5l"] Jul 14 18:48:29.222333 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:48:29.222298 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92e79146-dc86-42e9-9627-a39d5719aab9" path="/var/lib/kubelet/pods/92e79146-dc86-42e9-9627-a39d5719aab9/volumes" Jul 14 18:49:08.897499 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:08.897467 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf"] Jul 14 18:49:08.898102 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:08.897677 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" podUID="81c2c77a-f360-4c1e-a2d7-6a01232ebe0d" containerName="maas-api" containerID="cri-o://3ef12d1705da90dd9fd75aac285ca8494e1d55f4890d09d724e455ad7b054baa" gracePeriod=30 Jul 14 18:49:08.922834 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:08.922803 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:09.157864 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.157805 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" Jul 14 18:49:09.212434 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.212402 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-openshift-service-ca\") pod \"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d\" (UID: \"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d\") " Jul 14 18:49:09.212586 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.212447 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-maas-api-tls\") pod \"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d\" (UID: \"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d\") " Jul 14 18:49:09.212586 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.212516 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtgnc\" (UniqueName: \"kubernetes.io/projected/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-kube-api-access-dtgnc\") pod \"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d\" (UID: \"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d\") " Jul 14 18:49:09.212787 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.212760 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "81c2c77a-f360-4c1e-a2d7-6a01232ebe0d" (UID: "81c2c77a-f360-4c1e-a2d7-6a01232ebe0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:49:09.214587 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.214561 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-kube-api-access-dtgnc" (OuterVolumeSpecName: "kube-api-access-dtgnc") pod "81c2c77a-f360-4c1e-a2d7-6a01232ebe0d" (UID: "81c2c77a-f360-4c1e-a2d7-6a01232ebe0d"). InnerVolumeSpecName "kube-api-access-dtgnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:49:09.214587 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.214567 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "81c2c77a-f360-4c1e-a2d7-6a01232ebe0d" (UID: "81c2c77a-f360-4c1e-a2d7-6a01232ebe0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:49:09.313570 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.313536 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtgnc\" (UniqueName: \"kubernetes.io/projected/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-kube-api-access-dtgnc\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:49:09.313677 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.313572 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-openshift-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:49:09.313677 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.313587 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d-maas-api-tls\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:49:09.411714 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.411649 2573 generic.go:356] "Generic (PLEG): container finished" podID="81c2c77a-f360-4c1e-a2d7-6a01232ebe0d" containerID="3ef12d1705da90dd9fd75aac285ca8494e1d55f4890d09d724e455ad7b054baa" exitCode=0 Jul 14 18:49:09.411714 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.411697 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" Jul 14 18:49:09.411865 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.411744 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" event={"ID":"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d","Type":"ContainerDied","Data":"3ef12d1705da90dd9fd75aac285ca8494e1d55f4890d09d724e455ad7b054baa"} Jul 14 18:49:09.411865 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.411785 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf" event={"ID":"81c2c77a-f360-4c1e-a2d7-6a01232ebe0d","Type":"ContainerDied","Data":"c582e6cff427663baa41fe0d0f249e6868c467e4037c0962b318c1d128590b7d"} Jul 14 18:49:09.411865 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.411811 2573 scope.go:117] "RemoveContainer" containerID="3ef12d1705da90dd9fd75aac285ca8494e1d55f4890d09d724e455ad7b054baa" Jul 14 18:49:09.421088 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.421072 2573 scope.go:117] "RemoveContainer" containerID="3ef12d1705da90dd9fd75aac285ca8494e1d55f4890d09d724e455ad7b054baa" Jul 14 18:49:09.421373 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:49:09.421355 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef12d1705da90dd9fd75aac285ca8494e1d55f4890d09d724e455ad7b054baa\": container with ID starting with 3ef12d1705da90dd9fd75aac285ca8494e1d55f4890d09d724e455ad7b054baa not found: ID does not exist" containerID="3ef12d1705da90dd9fd75aac285ca8494e1d55f4890d09d724e455ad7b054baa" Jul 14 18:49:09.421437 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.421382 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef12d1705da90dd9fd75aac285ca8494e1d55f4890d09d724e455ad7b054baa"} err="failed to get container status \"3ef12d1705da90dd9fd75aac285ca8494e1d55f4890d09d724e455ad7b054baa\": rpc error: code = NotFound desc = could not find container \"3ef12d1705da90dd9fd75aac285ca8494e1d55f4890d09d724e455ad7b054baa\": container with ID starting with 3ef12d1705da90dd9fd75aac285ca8494e1d55f4890d09d724e455ad7b054baa not found: ID does not exist" Jul 14 18:49:09.430530 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.430504 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf"] Jul 14 18:49:09.432698 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:09.432679 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bfd487d2-5f9879fc4d-gj7jf"] Jul 14 18:49:11.223077 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:11.223044 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81c2c77a-f360-4c1e-a2d7-6a01232ebe0d" path="/var/lib/kubelet/pods/81c2c77a-f360-4c1e-a2d7-6a01232ebe0d/volumes" Jul 14 18:49:13.546346 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:13.546314 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:19.858997 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:19.858967 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:19.959018 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:19.958988 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:23.744184 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:23.744142 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:24.125822 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:24.125735 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:24.225834 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:24.225798 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:24.341161 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:24.341119 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:25.564818 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.564774 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:25.789098 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.789064 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z"] Jul 14 18:49:25.789656 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.789636 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92e79146-dc86-42e9-9627-a39d5719aab9" containerName="revoke-keys" Jul 14 18:49:25.789656 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.789657 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="92e79146-dc86-42e9-9627-a39d5719aab9" containerName="revoke-keys" Jul 14 18:49:25.789836 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.789679 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92e79146-dc86-42e9-9627-a39d5719aab9" containerName="revoke-keys" Jul 14 18:49:25.789836 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.789687 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="92e79146-dc86-42e9-9627-a39d5719aab9" containerName="revoke-keys" Jul 14 18:49:25.789836 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.789704 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92e79146-dc86-42e9-9627-a39d5719aab9" containerName="revoke-keys" Jul 14 18:49:25.789836 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.789714 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="92e79146-dc86-42e9-9627-a39d5719aab9" containerName="revoke-keys" Jul 14 18:49:25.789836 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.789735 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81c2c77a-f360-4c1e-a2d7-6a01232ebe0d" containerName="maas-api" Jul 14 18:49:25.789836 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.789742 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="81c2c77a-f360-4c1e-a2d7-6a01232ebe0d" containerName="maas-api" Jul 14 18:49:25.790127 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.789862 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="92e79146-dc86-42e9-9627-a39d5719aab9" containerName="revoke-keys" Jul 14 18:49:25.790127 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.789879 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="81c2c77a-f360-4c1e-a2d7-6a01232ebe0d" containerName="maas-api" Jul 14 18:49:25.790127 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.789890 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="92e79146-dc86-42e9-9627-a39d5719aab9" containerName="revoke-keys" Jul 14 18:49:25.793928 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.793906 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:49:25.796810 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.796788 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-541ed9bd-serving-cert\"" Jul 14 18:49:25.804953 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.804927 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z"] Jul 14 18:49:25.825075 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.825003 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:25.867214 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.867192 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d34600-9882-42a9-8ac2-a2daeac5801b-openshift-service-ca\") pod \"maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z\" (UID: \"24d34600-9882-42a9-8ac2-a2daeac5801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:49:25.867363 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.867227 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c26lx\" (UniqueName: \"kubernetes.io/projected/24d34600-9882-42a9-8ac2-a2daeac5801b-kube-api-access-c26lx\") pod \"maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z\" (UID: \"24d34600-9882-42a9-8ac2-a2daeac5801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:49:25.867363 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.867330 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24d34600-9882-42a9-8ac2-a2daeac5801b-maas-api-tls\") pod \"maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z\" (UID: \"24d34600-9882-42a9-8ac2-a2daeac5801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:49:25.968524 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.968499 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c26lx\" (UniqueName: \"kubernetes.io/projected/24d34600-9882-42a9-8ac2-a2daeac5801b-kube-api-access-c26lx\") pod \"maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z\" (UID: \"24d34600-9882-42a9-8ac2-a2daeac5801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:49:25.968658 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.968572 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24d34600-9882-42a9-8ac2-a2daeac5801b-maas-api-tls\") pod \"maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z\" (UID: \"24d34600-9882-42a9-8ac2-a2daeac5801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:49:25.968705 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.968693 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d34600-9882-42a9-8ac2-a2daeac5801b-openshift-service-ca\") pod \"maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z\" (UID: \"24d34600-9882-42a9-8ac2-a2daeac5801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:49:25.968741 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:49:25.968724 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-serving-cert: secret "maas-api-e2e-migrate-541ed9bd-serving-cert" not found Jul 14 18:49:25.968824 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:49:25.968813 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24d34600-9882-42a9-8ac2-a2daeac5801b-maas-api-tls podName:24d34600-9882-42a9-8ac2-a2daeac5801b nodeName:}" failed. No retries permitted until 2026-07-14 18:49:26.468790427 +0000 UTC m=+1668.006238207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24d34600-9882-42a9-8ac2-a2daeac5801b-maas-api-tls") pod "maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" (UID: "24d34600-9882-42a9-8ac2-a2daeac5801b") : secret "maas-api-e2e-migrate-541ed9bd-serving-cert" not found Jul 14 18:49:25.969332 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.969315 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d34600-9882-42a9-8ac2-a2daeac5801b-openshift-service-ca\") pod \"maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z\" (UID: \"24d34600-9882-42a9-8ac2-a2daeac5801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:49:25.979849 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:25.979821 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c26lx\" (UniqueName: \"kubernetes.io/projected/24d34600-9882-42a9-8ac2-a2daeac5801b-kube-api-access-c26lx\") pod \"maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z\" (UID: \"24d34600-9882-42a9-8ac2-a2daeac5801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:49:26.106636 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:26.106542 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:26.266782 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:26.266739 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:26.367178 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:26.367086 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:26.473535 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:26.473498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24d34600-9882-42a9-8ac2-a2daeac5801b-maas-api-tls\") pod \"maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z\" (UID: \"24d34600-9882-42a9-8ac2-a2daeac5801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:49:26.476074 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:26.476005 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24d34600-9882-42a9-8ac2-a2daeac5801b-maas-api-tls\") pod \"maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z\" (UID: \"24d34600-9882-42a9-8ac2-a2daeac5801b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:49:26.708041 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:26.708014 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:49:26.852221 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:26.852193 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z"] Jul 14 18:49:26.853972 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:49:26.853939 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d34600_9882_42a9_8ac2_a2daeac5801b.slice/crio-460d42d4b8c14bf9bf2dd36b14ff7f56f06682feb2a9359096c380b15dd17528 WatchSource:0}: Error finding container 460d42d4b8c14bf9bf2dd36b14ff7f56f06682feb2a9359096c380b15dd17528: Status 404 returned error can't find the container with id 460d42d4b8c14bf9bf2dd36b14ff7f56f06682feb2a9359096c380b15dd17528 Jul 14 18:49:27.520217 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:27.520184 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" event={"ID":"24d34600-9882-42a9-8ac2-a2daeac5801b","Type":"ContainerStarted","Data":"79e3e086a40fc004b3cb6a86e8abc7f222573b4d02aea6d039855e90f1929355"} Jul 14 18:49:27.520382 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:27.520242 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" event={"ID":"24d34600-9882-42a9-8ac2-a2daeac5801b","Type":"ContainerStarted","Data":"460d42d4b8c14bf9bf2dd36b14ff7f56f06682feb2a9359096c380b15dd17528"} Jul 14 18:49:27.520461 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:27.520439 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:49:27.537900 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:27.537835 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" podStartSLOduration=2.051666161 podStartE2EDuration="2.537819727s" totalImagesPullingTime="486.153566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:49:27.536507537 +0000 UTC m=+1669.073955322" watchObservedRunningTime="2026-07-14 18:49:27.537819727 +0000 UTC m=+1669.075267513" Jul 14 18:49:30.106314 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:30.106261 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:30.167726 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:30.167670 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:33.532649 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:33.532618 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:49:33.699738 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:33.699701 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:53.135031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:53.134994 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:49:53.236843 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:49:53.236801 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:30.337608 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:30.337569 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:30.686707 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:30.686622 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z"] Jul 14 18:50:30.687198 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:30.687161 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" podUID="24d34600-9882-42a9-8ac2-a2daeac5801b" containerName="maas-api" containerID="cri-o://79e3e086a40fc004b3cb6a86e8abc7f222573b4d02aea6d039855e90f1929355" gracePeriod=30 Jul 14 18:50:30.894778 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:30.894738 2573 generic.go:356] "Generic (PLEG): container finished" podID="24d34600-9882-42a9-8ac2-a2daeac5801b" containerID="79e3e086a40fc004b3cb6a86e8abc7f222573b4d02aea6d039855e90f1929355" exitCode=0 Jul 14 18:50:30.894929 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:30.894812 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" event={"ID":"24d34600-9882-42a9-8ac2-a2daeac5801b","Type":"ContainerDied","Data":"79e3e086a40fc004b3cb6a86e8abc7f222573b4d02aea6d039855e90f1929355"} Jul 14 18:50:30.950927 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:30.950906 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:50:30.979190 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:30.979164 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c26lx\" (UniqueName: \"kubernetes.io/projected/24d34600-9882-42a9-8ac2-a2daeac5801b-kube-api-access-c26lx\") pod \"24d34600-9882-42a9-8ac2-a2daeac5801b\" (UID: \"24d34600-9882-42a9-8ac2-a2daeac5801b\") " Jul 14 18:50:30.979310 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:30.979205 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24d34600-9882-42a9-8ac2-a2daeac5801b-maas-api-tls\") pod \"24d34600-9882-42a9-8ac2-a2daeac5801b\" (UID: \"24d34600-9882-42a9-8ac2-a2daeac5801b\") " Jul 14 18:50:30.979310 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:30.979281 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d34600-9882-42a9-8ac2-a2daeac5801b-openshift-service-ca\") pod \"24d34600-9882-42a9-8ac2-a2daeac5801b\" (UID: \"24d34600-9882-42a9-8ac2-a2daeac5801b\") " Jul 14 18:50:30.979626 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:30.979604 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d34600-9882-42a9-8ac2-a2daeac5801b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24d34600-9882-42a9-8ac2-a2daeac5801b" (UID: "24d34600-9882-42a9-8ac2-a2daeac5801b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:50:30.981262 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:30.981246 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d34600-9882-42a9-8ac2-a2daeac5801b-kube-api-access-c26lx" (OuterVolumeSpecName: "kube-api-access-c26lx") pod "24d34600-9882-42a9-8ac2-a2daeac5801b" (UID: "24d34600-9882-42a9-8ac2-a2daeac5801b"). InnerVolumeSpecName "kube-api-access-c26lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:50:30.981406 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:30.981380 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d34600-9882-42a9-8ac2-a2daeac5801b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24d34600-9882-42a9-8ac2-a2daeac5801b" (UID: "24d34600-9882-42a9-8ac2-a2daeac5801b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:50:31.080809 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:31.080780 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d34600-9882-42a9-8ac2-a2daeac5801b-openshift-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:50:31.080809 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:31.080804 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c26lx\" (UniqueName: \"kubernetes.io/projected/24d34600-9882-42a9-8ac2-a2daeac5801b-kube-api-access-c26lx\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:50:31.080809 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:31.080815 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24d34600-9882-42a9-8ac2-a2daeac5801b-maas-api-tls\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:50:31.651735 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:31.651637 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:31.902788 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:31.902698 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" Jul 14 18:50:31.902788 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:31.902743 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z" event={"ID":"24d34600-9882-42a9-8ac2-a2daeac5801b","Type":"ContainerDied","Data":"460d42d4b8c14bf9bf2dd36b14ff7f56f06682feb2a9359096c380b15dd17528"} Jul 14 18:50:31.903012 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:31.902807 2573 scope.go:117] "RemoveContainer" containerID="79e3e086a40fc004b3cb6a86e8abc7f222573b4d02aea6d039855e90f1929355" Jul 14 18:50:31.922537 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:31.922504 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z"] Jul 14 18:50:31.924821 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:31.924802 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-541ed9bd-5b4d4f7d88-tvd4z"] Jul 14 18:50:33.228885 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:33.228852 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24d34600-9882-42a9-8ac2-a2daeac5801b" path="/var/lib/kubelet/pods/24d34600-9882-42a9-8ac2-a2daeac5801b/volumes" Jul 14 18:50:37.007171 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:37.007131 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:37.161003 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:37.160959 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:37.263807 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:37.263706 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:37.396999 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:37.396964 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:37.491248 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:37.491216 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:38.583600 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.583564 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:38.674036 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.674006 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2"] Jul 14 18:50:38.674704 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.674685 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24d34600-9882-42a9-8ac2-a2daeac5801b" containerName="maas-api" Jul 14 18:50:38.674767 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.674708 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d34600-9882-42a9-8ac2-a2daeac5801b" containerName="maas-api" Jul 14 18:50:38.674850 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.674839 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="24d34600-9882-42a9-8ac2-a2daeac5801b" containerName="maas-api" Jul 14 18:50:38.674885 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.674854 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="92e79146-dc86-42e9-9627-a39d5719aab9" containerName="revoke-keys" Jul 14 18:50:38.679841 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.679819 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" Jul 14 18:50:38.682904 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.682880 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-01bc759f-serving-cert\"" Jul 14 18:50:38.687975 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.687946 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:38.690354 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.690334 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2"] Jul 14 18:50:38.756714 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.756684 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7537c3f8-209a-4725-9d6f-8759bc92ca4d-openshift-service-ca\") pod \"maas-api-e2e-ait-01bc759f-79d8686864-l7gs2\" (UID: \"7537c3f8-209a-4725-9d6f-8759bc92ca4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" Jul 14 18:50:38.756917 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.756740 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7537c3f8-209a-4725-9d6f-8759bc92ca4d-maas-api-tls\") pod \"maas-api-e2e-ait-01bc759f-79d8686864-l7gs2\" (UID: \"7537c3f8-209a-4725-9d6f-8759bc92ca4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" Jul 14 18:50:38.756917 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.756828 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb8wj\" (UniqueName: \"kubernetes.io/projected/7537c3f8-209a-4725-9d6f-8759bc92ca4d-kube-api-access-wb8wj\") pod \"maas-api-e2e-ait-01bc759f-79d8686864-l7gs2\" (UID: \"7537c3f8-209a-4725-9d6f-8759bc92ca4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" Jul 14 18:50:38.799479 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.799442 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:38.857575 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.857477 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7537c3f8-209a-4725-9d6f-8759bc92ca4d-openshift-service-ca\") pod \"maas-api-e2e-ait-01bc759f-79d8686864-l7gs2\" (UID: \"7537c3f8-209a-4725-9d6f-8759bc92ca4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" Jul 14 18:50:38.857575 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.857552 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7537c3f8-209a-4725-9d6f-8759bc92ca4d-maas-api-tls\") pod \"maas-api-e2e-ait-01bc759f-79d8686864-l7gs2\" (UID: \"7537c3f8-209a-4725-9d6f-8759bc92ca4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" Jul 14 18:50:38.857819 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.857605 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb8wj\" (UniqueName: \"kubernetes.io/projected/7537c3f8-209a-4725-9d6f-8759bc92ca4d-kube-api-access-wb8wj\") pod \"maas-api-e2e-ait-01bc759f-79d8686864-l7gs2\" (UID: \"7537c3f8-209a-4725-9d6f-8759bc92ca4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" Jul 14 18:50:38.858277 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.858256 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7537c3f8-209a-4725-9d6f-8759bc92ca4d-openshift-service-ca\") pod \"maas-api-e2e-ait-01bc759f-79d8686864-l7gs2\" (UID: \"7537c3f8-209a-4725-9d6f-8759bc92ca4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" Jul 14 18:50:38.862650 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.862630 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-01bc759f-serving-cert\"" Jul 14 18:50:38.870905 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.870883 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7537c3f8-209a-4725-9d6f-8759bc92ca4d-maas-api-tls\") pod \"maas-api-e2e-ait-01bc759f-79d8686864-l7gs2\" (UID: \"7537c3f8-209a-4725-9d6f-8759bc92ca4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" Jul 14 18:50:38.876632 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.876607 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb8wj\" (UniqueName: \"kubernetes.io/projected/7537c3f8-209a-4725-9d6f-8759bc92ca4d-kube-api-access-wb8wj\") pod \"maas-api-e2e-ait-01bc759f-79d8686864-l7gs2\" (UID: \"7537c3f8-209a-4725-9d6f-8759bc92ca4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" Jul 14 18:50:38.897377 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.897346 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:38.993008 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:38.992964 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" Jul 14 18:50:39.333577 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:39.333536 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:39.348346 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:39.348321 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2"] Jul 14 18:50:39.350483 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:50:39.350458 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7537c3f8_209a_4725_9d6f_8759bc92ca4d.slice/crio-19ae4d9d148c7637000437b4e671c40cf63f703dd954a922b940b68ece87467a WatchSource:0}: Error finding container 19ae4d9d148c7637000437b4e671c40cf63f703dd954a922b940b68ece87467a: Status 404 returned error can't find the container with id 19ae4d9d148c7637000437b4e671c40cf63f703dd954a922b940b68ece87467a Jul 14 18:50:39.352172 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:39.352139 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:50:39.622570 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:39.622473 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:39.957197 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:39.957162 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" event={"ID":"7537c3f8-209a-4725-9d6f-8759bc92ca4d","Type":"ContainerStarted","Data":"19ae4d9d148c7637000437b4e671c40cf63f703dd954a922b940b68ece87467a"} Jul 14 18:50:40.756576 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:40.756542 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:50:40.965609 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:40.965570 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" event={"ID":"7537c3f8-209a-4725-9d6f-8759bc92ca4d","Type":"ContainerStarted","Data":"6dc504936b935067294aba5043175359541924811d1fe7485712479071ea7c2c"} Jul 14 18:50:40.965776 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:40.965637 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" Jul 14 18:50:40.984780 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:40.984697 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" podStartSLOduration=1.5726260779999999 podStartE2EDuration="2.984682111s" totalImagesPullingTime="1.412056033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:50:40.982566082 +0000 UTC m=+1742.520013866" watchObservedRunningTime="2026-07-14 18:50:40.984682111 +0000 UTC m=+1742.522129896" Jul 14 18:50:46.976866 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:46.976836 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" Jul 14 18:50:47.465820 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:50:47.465786 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:51:09.033960 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:09.033920 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:51:09.131807 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:09.131764 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:51:34.366625 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:34.366546 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:51:34.466501 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:34.466464 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:51:39.010153 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:39.010123 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:51:39.014193 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:39.014171 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:51:39.017894 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:39.017877 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:51:39.021797 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:39.021779 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:51:44.328487 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:44.328451 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2"] Jul 14 18:51:44.329351 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:44.328742 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" podUID="7537c3f8-209a-4725-9d6f-8759bc92ca4d" containerName="maas-api" containerID="cri-o://6dc504936b935067294aba5043175359541924811d1fe7485712479071ea7c2c" gracePeriod=30 Jul 14 18:51:44.360408 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:44.360376 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:51:44.589779 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:44.589709 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" Jul 14 18:51:44.690689 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:44.690658 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7537c3f8-209a-4725-9d6f-8759bc92ca4d-openshift-service-ca\") pod \"7537c3f8-209a-4725-9d6f-8759bc92ca4d\" (UID: \"7537c3f8-209a-4725-9d6f-8759bc92ca4d\") " Jul 14 18:51:44.690879 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:44.690720 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb8wj\" (UniqueName: \"kubernetes.io/projected/7537c3f8-209a-4725-9d6f-8759bc92ca4d-kube-api-access-wb8wj\") pod \"7537c3f8-209a-4725-9d6f-8759bc92ca4d\" (UID: \"7537c3f8-209a-4725-9d6f-8759bc92ca4d\") " Jul 14 18:51:44.690879 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:44.690790 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7537c3f8-209a-4725-9d6f-8759bc92ca4d-maas-api-tls\") pod \"7537c3f8-209a-4725-9d6f-8759bc92ca4d\" (UID: \"7537c3f8-209a-4725-9d6f-8759bc92ca4d\") " Jul 14 18:51:44.691023 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:44.690997 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7537c3f8-209a-4725-9d6f-8759bc92ca4d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7537c3f8-209a-4725-9d6f-8759bc92ca4d" (UID: "7537c3f8-209a-4725-9d6f-8759bc92ca4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:51:44.691188 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:44.691171 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7537c3f8-209a-4725-9d6f-8759bc92ca4d-openshift-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:51:44.692924 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:44.692898 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7537c3f8-209a-4725-9d6f-8759bc92ca4d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7537c3f8-209a-4725-9d6f-8759bc92ca4d" (UID: "7537c3f8-209a-4725-9d6f-8759bc92ca4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:51:44.693064 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:44.692967 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7537c3f8-209a-4725-9d6f-8759bc92ca4d-kube-api-access-wb8wj" (OuterVolumeSpecName: "kube-api-access-wb8wj") pod "7537c3f8-209a-4725-9d6f-8759bc92ca4d" (UID: "7537c3f8-209a-4725-9d6f-8759bc92ca4d"). InnerVolumeSpecName "kube-api-access-wb8wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:51:44.791750 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:44.791722 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7537c3f8-209a-4725-9d6f-8759bc92ca4d-maas-api-tls\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:51:44.791750 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:44.791748 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb8wj\" (UniqueName: \"kubernetes.io/projected/7537c3f8-209a-4725-9d6f-8759bc92ca4d-kube-api-access-wb8wj\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:51:45.340000 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:45.339968 2573 generic.go:356] "Generic (PLEG): container finished" podID="7537c3f8-209a-4725-9d6f-8759bc92ca4d" containerID="6dc504936b935067294aba5043175359541924811d1fe7485712479071ea7c2c" exitCode=0 Jul 14 18:51:45.340417 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:45.340006 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" Jul 14 18:51:45.340417 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:45.340053 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" event={"ID":"7537c3f8-209a-4725-9d6f-8759bc92ca4d","Type":"ContainerDied","Data":"6dc504936b935067294aba5043175359541924811d1fe7485712479071ea7c2c"} Jul 14 18:51:45.340417 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:45.340104 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2" event={"ID":"7537c3f8-209a-4725-9d6f-8759bc92ca4d","Type":"ContainerDied","Data":"19ae4d9d148c7637000437b4e671c40cf63f703dd954a922b940b68ece87467a"} Jul 14 18:51:45.340417 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:45.340127 2573 scope.go:117] "RemoveContainer" containerID="6dc504936b935067294aba5043175359541924811d1fe7485712479071ea7c2c" Jul 14 18:51:45.349682 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:45.349664 2573 scope.go:117] "RemoveContainer" containerID="6dc504936b935067294aba5043175359541924811d1fe7485712479071ea7c2c" Jul 14 18:51:45.349969 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:51:45.349950 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc504936b935067294aba5043175359541924811d1fe7485712479071ea7c2c\": container with ID starting with 6dc504936b935067294aba5043175359541924811d1fe7485712479071ea7c2c not found: ID does not exist" containerID="6dc504936b935067294aba5043175359541924811d1fe7485712479071ea7c2c" Jul 14 18:51:45.350031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:45.349977 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc504936b935067294aba5043175359541924811d1fe7485712479071ea7c2c"} err="failed to get container status \"6dc504936b935067294aba5043175359541924811d1fe7485712479071ea7c2c\": rpc error: code = NotFound desc = could not find container \"6dc504936b935067294aba5043175359541924811d1fe7485712479071ea7c2c\": container with ID starting with 6dc504936b935067294aba5043175359541924811d1fe7485712479071ea7c2c not found: ID does not exist" Jul 14 18:51:45.361770 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:45.361746 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2"] Jul 14 18:51:45.369010 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:45.368991 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-01bc759f-79d8686864-l7gs2"] Jul 14 18:51:47.223728 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:47.223696 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7537c3f8-209a-4725-9d6f-8759bc92ca4d" path="/var/lib/kubelet/pods/7537c3f8-209a-4725-9d6f-8759bc92ca4d/volumes" Jul 14 18:51:48.964031 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:48.964002 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:51:55.002142 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:55.002106 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:51:55.103032 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:51:55.102993 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:52:00.256861 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:00.256828 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:52:00.642008 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:00.641964 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:52:01.767496 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:01.767445 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:52:01.931094 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:01.931051 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:52:02.033695 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.033617 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779"] Jul 14 18:52:02.034111 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.034098 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7537c3f8-209a-4725-9d6f-8759bc92ca4d" containerName="maas-api" Jul 14 18:52:02.034111 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.034112 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7537c3f8-209a-4725-9d6f-8759bc92ca4d" containerName="maas-api" Jul 14 18:52:02.034196 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.034187 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7537c3f8-209a-4725-9d6f-8759bc92ca4d" containerName="maas-api" Jul 14 18:52:02.037568 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.037553 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" Jul 14 18:52:02.050710 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.050681 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a8b422c9-serving-cert\"" Jul 14 18:52:02.056153 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.056131 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779"] Jul 14 18:52:02.091023 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.090994 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:52:02.152295 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.152258 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf4zr\" (UniqueName: \"kubernetes.io/projected/fc9594a9-fd11-48c0-ad21-b4780f991eef-kube-api-access-jf4zr\") pod \"maas-api-e2e-derive-a8b422c9-658458bb5f-6b779\" (UID: \"fc9594a9-fd11-48c0-ad21-b4780f991eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" Jul 14 18:52:02.152421 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.152329 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9594a9-fd11-48c0-ad21-b4780f991eef-maas-api-tls\") pod \"maas-api-e2e-derive-a8b422c9-658458bb5f-6b779\" (UID: \"fc9594a9-fd11-48c0-ad21-b4780f991eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" Jul 14 18:52:02.152468 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.152433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc9594a9-fd11-48c0-ad21-b4780f991eef-openshift-service-ca\") pod \"maas-api-e2e-derive-a8b422c9-658458bb5f-6b779\" (UID: \"fc9594a9-fd11-48c0-ad21-b4780f991eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" Jul 14 18:52:02.191102 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.191071 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:52:02.253030 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.252998 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9594a9-fd11-48c0-ad21-b4780f991eef-maas-api-tls\") pod \"maas-api-e2e-derive-a8b422c9-658458bb5f-6b779\" (UID: \"fc9594a9-fd11-48c0-ad21-b4780f991eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" Jul 14 18:52:02.253159 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.253065 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc9594a9-fd11-48c0-ad21-b4780f991eef-openshift-service-ca\") pod \"maas-api-e2e-derive-a8b422c9-658458bb5f-6b779\" (UID: \"fc9594a9-fd11-48c0-ad21-b4780f991eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" Jul 14 18:52:02.253159 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.253117 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf4zr\" (UniqueName: \"kubernetes.io/projected/fc9594a9-fd11-48c0-ad21-b4780f991eef-kube-api-access-jf4zr\") pod \"maas-api-e2e-derive-a8b422c9-658458bb5f-6b779\" (UID: \"fc9594a9-fd11-48c0-ad21-b4780f991eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" Jul 14 18:52:02.253850 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.253830 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc9594a9-fd11-48c0-ad21-b4780f991eef-openshift-service-ca\") pod \"maas-api-e2e-derive-a8b422c9-658458bb5f-6b779\" (UID: \"fc9594a9-fd11-48c0-ad21-b4780f991eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" Jul 14 18:52:02.255426 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.255405 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9594a9-fd11-48c0-ad21-b4780f991eef-maas-api-tls\") pod \"maas-api-e2e-derive-a8b422c9-658458bb5f-6b779\" (UID: \"fc9594a9-fd11-48c0-ad21-b4780f991eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" Jul 14 18:52:02.261770 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.261749 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf4zr\" (UniqueName: \"kubernetes.io/projected/fc9594a9-fd11-48c0-ad21-b4780f991eef-kube-api-access-jf4zr\") pod \"maas-api-e2e-derive-a8b422c9-658458bb5f-6b779\" (UID: \"fc9594a9-fd11-48c0-ad21-b4780f991eef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" Jul 14 18:52:02.354895 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.354836 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" Jul 14 18:52:02.498662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.498624 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779"] Jul 14 18:52:02.501866 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:52:02.501833 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9594a9_fd11_48c0_ad21_b4780f991eef.slice/crio-a618c1344cd62a92798802fc0ec0cbe65763e62d82fbaa6d0bf39ccc9fb81b84 WatchSource:0}: Error finding container a618c1344cd62a92798802fc0ec0cbe65763e62d82fbaa6d0bf39ccc9fb81b84: Status 404 returned error can't find the container with id a618c1344cd62a92798802fc0ec0cbe65763e62d82fbaa6d0bf39ccc9fb81b84 Jul 14 18:52:02.615228 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.615139 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:52:02.711165 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:02.711134 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:52:03.444691 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:03.444662 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" event={"ID":"fc9594a9-fd11-48c0-ad21-b4780f991eef","Type":"ContainerStarted","Data":"eaa777fe07877c6e0a5b21c7972072d4af07aa45f099bf0283d828d53dde0edf"} Jul 14 18:52:03.445045 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:03.444713 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" event={"ID":"fc9594a9-fd11-48c0-ad21-b4780f991eef","Type":"ContainerStarted","Data":"a618c1344cd62a92798802fc0ec0cbe65763e62d82fbaa6d0bf39ccc9fb81b84"} Jul 14 18:52:03.445045 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:03.444859 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" Jul 14 18:52:03.462238 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:03.462198 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" podStartSLOduration=1.67020832 podStartE2EDuration="2.462184584s" totalImagesPullingTime="791.976264ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:52:03.46178548 +0000 UTC m=+1824.999233267" watchObservedRunningTime="2026-07-14 18:52:03.462184584 +0000 UTC m=+1824.999632434" Jul 14 18:52:06.761171 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:06.761136 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:52:06.855541 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:06.855506 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:52:09.456060 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:09.456029 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" Jul 14 18:52:10.065449 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:52:10.065411 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:16.905518 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:16.905480 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779"] Jul 14 18:53:16.906080 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:16.905766 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" podUID="fc9594a9-fd11-48c0-ad21-b4780f991eef" containerName="maas-api" containerID="cri-o://eaa777fe07877c6e0a5b21c7972072d4af07aa45f099bf0283d828d53dde0edf" gracePeriod=30 Jul 14 18:53:16.934589 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:16.934544 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:17.170429 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.170366 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" Jul 14 18:53:17.285464 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.285432 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc9594a9-fd11-48c0-ad21-b4780f991eef-openshift-service-ca\") pod \"fc9594a9-fd11-48c0-ad21-b4780f991eef\" (UID: \"fc9594a9-fd11-48c0-ad21-b4780f991eef\") " Jul 14 18:53:17.285653 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.285477 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf4zr\" (UniqueName: \"kubernetes.io/projected/fc9594a9-fd11-48c0-ad21-b4780f991eef-kube-api-access-jf4zr\") pod \"fc9594a9-fd11-48c0-ad21-b4780f991eef\" (UID: \"fc9594a9-fd11-48c0-ad21-b4780f991eef\") " Jul 14 18:53:17.285653 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.285537 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9594a9-fd11-48c0-ad21-b4780f991eef-maas-api-tls\") pod \"fc9594a9-fd11-48c0-ad21-b4780f991eef\" (UID: \"fc9594a9-fd11-48c0-ad21-b4780f991eef\") " Jul 14 18:53:17.285878 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.285851 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9594a9-fd11-48c0-ad21-b4780f991eef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fc9594a9-fd11-48c0-ad21-b4780f991eef" (UID: "fc9594a9-fd11-48c0-ad21-b4780f991eef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:53:17.286023 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.286003 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc9594a9-fd11-48c0-ad21-b4780f991eef-openshift-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:53:17.287663 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.287638 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc9594a9-fd11-48c0-ad21-b4780f991eef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fc9594a9-fd11-48c0-ad21-b4780f991eef" (UID: "fc9594a9-fd11-48c0-ad21-b4780f991eef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:53:17.287663 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.287644 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9594a9-fd11-48c0-ad21-b4780f991eef-kube-api-access-jf4zr" (OuterVolumeSpecName: "kube-api-access-jf4zr") pod "fc9594a9-fd11-48c0-ad21-b4780f991eef" (UID: "fc9594a9-fd11-48c0-ad21-b4780f991eef"). InnerVolumeSpecName "kube-api-access-jf4zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:53:17.387590 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.387554 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9594a9-fd11-48c0-ad21-b4780f991eef-maas-api-tls\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:53:17.387590 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.387584 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf4zr\" (UniqueName: \"kubernetes.io/projected/fc9594a9-fd11-48c0-ad21-b4780f991eef-kube-api-access-jf4zr\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:53:17.880469 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.880432 2573 generic.go:356] "Generic (PLEG): container finished" podID="fc9594a9-fd11-48c0-ad21-b4780f991eef" containerID="eaa777fe07877c6e0a5b21c7972072d4af07aa45f099bf0283d828d53dde0edf" exitCode=0 Jul 14 18:53:17.880643 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.880479 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" Jul 14 18:53:17.880643 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.880513 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" event={"ID":"fc9594a9-fd11-48c0-ad21-b4780f991eef","Type":"ContainerDied","Data":"eaa777fe07877c6e0a5b21c7972072d4af07aa45f099bf0283d828d53dde0edf"} Jul 14 18:53:17.880643 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.880567 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779" event={"ID":"fc9594a9-fd11-48c0-ad21-b4780f991eef","Type":"ContainerDied","Data":"a618c1344cd62a92798802fc0ec0cbe65763e62d82fbaa6d0bf39ccc9fb81b84"} Jul 14 18:53:17.880643 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.880591 2573 scope.go:117] "RemoveContainer" containerID="eaa777fe07877c6e0a5b21c7972072d4af07aa45f099bf0283d828d53dde0edf" Jul 14 18:53:17.892933 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.892914 2573 scope.go:117] "RemoveContainer" containerID="eaa777fe07877c6e0a5b21c7972072d4af07aa45f099bf0283d828d53dde0edf" Jul 14 18:53:17.893209 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:53:17.893189 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa777fe07877c6e0a5b21c7972072d4af07aa45f099bf0283d828d53dde0edf\": container with ID starting with eaa777fe07877c6e0a5b21c7972072d4af07aa45f099bf0283d828d53dde0edf not found: ID does not exist" containerID="eaa777fe07877c6e0a5b21c7972072d4af07aa45f099bf0283d828d53dde0edf" Jul 14 18:53:17.893279 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.893223 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa777fe07877c6e0a5b21c7972072d4af07aa45f099bf0283d828d53dde0edf"} err="failed to get container status \"eaa777fe07877c6e0a5b21c7972072d4af07aa45f099bf0283d828d53dde0edf\": rpc error: code = NotFound desc = could not find container \"eaa777fe07877c6e0a5b21c7972072d4af07aa45f099bf0283d828d53dde0edf\": container with ID starting with eaa777fe07877c6e0a5b21c7972072d4af07aa45f099bf0283d828d53dde0edf not found: ID does not exist" Jul 14 18:53:17.907354 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.907330 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779"] Jul 14 18:53:17.910886 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:17.910866 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a8b422c9-658458bb5f-6b779"] Jul 14 18:53:19.224351 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:19.224317 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc9594a9-fd11-48c0-ad21-b4780f991eef" path="/var/lib/kubelet/pods/fc9594a9-fd11-48c0-ad21-b4780f991eef/volumes" Jul 14 18:53:21.423684 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:21.423647 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:27.620654 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:27.620619 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:27.684887 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:27.684852 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:31.642486 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:31.642454 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:32.337062 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:32.337017 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:32.438785 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:32.438751 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:32.583413 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:32.583378 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:34.259860 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:34.259820 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:38.279418 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:38.279388 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:40.320458 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:40.320422 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:40.381344 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:40.381313 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:40.472748 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:40.472707 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:40.572865 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:40.572766 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:40.673520 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:40.673484 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:53:41.186015 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:53:41.185975 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:07.306032 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:07.305999 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:14.098537 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:14.098498 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:14.211810 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:14.211768 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:14.387536 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:14.387443 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:30.411544 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:30.411500 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:36.582634 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:36.582581 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:36.649166 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:36.649131 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:36.745925 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:36.745864 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:36.840785 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:36.840686 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:39.108519 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:39.108492 2573 scope.go:117] "RemoveContainer" containerID="b08058ab152f2e670db2865149d17ec2e479844a48f3b8f2ec499783f22f76fe" Jul 14 18:54:47.498045 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:47.498005 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:48.439679 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:48.439647 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:55.210356 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:55.210320 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:55.296126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:55.296091 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:55.437159 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:55.437120 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:55.562506 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:55.562419 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:56.579959 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:56.579925 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:56.948543 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:56.948511 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:54:58.851429 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:54:58.851394 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:05.169011 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:05.168949 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:05.268699 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:05.268661 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:05.948419 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:05.948381 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:12.162707 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:12.162596 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:12.267257 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:12.267226 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:12.507937 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:12.507893 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:12.839135 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:12.839046 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:12.991466 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:12.991421 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:16.162004 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.161920 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:16.244550 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.244516 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g"] Jul 14 18:55:16.245334 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.245314 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9594a9-fd11-48c0-ad21-b4780f991eef" containerName="maas-api" Jul 14 18:55:16.245392 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.245338 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9594a9-fd11-48c0-ad21-b4780f991eef" containerName="maas-api" Jul 14 18:55:16.245480 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.245467 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc9594a9-fd11-48c0-ad21-b4780f991eef" containerName="maas-api" Jul 14 18:55:16.249276 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.249256 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:16.252044 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.252023 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-34c47f4d-serving-cert\"" Jul 14 18:55:16.255741 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.255716 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g"] Jul 14 18:55:16.321376 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.321343 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff38d09-d990-491f-b780-819f4c390998-openshift-service-ca\") pod \"maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g\" (UID: \"fff38d09-d990-491f-b780-819f4c390998\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:16.321543 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.321417 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff38d09-d990-491f-b780-819f4c390998-maas-api-tls\") pod \"maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g\" (UID: \"fff38d09-d990-491f-b780-819f4c390998\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:16.321617 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.321595 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z5rq\" (UniqueName: \"kubernetes.io/projected/fff38d09-d990-491f-b780-819f4c390998-kube-api-access-5z5rq\") pod \"maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g\" (UID: \"fff38d09-d990-491f-b780-819f4c390998\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:16.422623 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.422527 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff38d09-d990-491f-b780-819f4c390998-openshift-service-ca\") pod \"maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g\" (UID: \"fff38d09-d990-491f-b780-819f4c390998\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:16.422623 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.422588 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff38d09-d990-491f-b780-819f4c390998-maas-api-tls\") pod \"maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g\" (UID: \"fff38d09-d990-491f-b780-819f4c390998\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:16.422856 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.422648 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z5rq\" (UniqueName: \"kubernetes.io/projected/fff38d09-d990-491f-b780-819f4c390998-kube-api-access-5z5rq\") pod \"maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g\" (UID: \"fff38d09-d990-491f-b780-819f4c390998\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:16.422856 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:55:16.422759 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-serving-cert: secret "maas-api-e2e-mt-34c47f4d-serving-cert" not found Jul 14 18:55:16.422856 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:55:16.422825 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fff38d09-d990-491f-b780-819f4c390998-maas-api-tls podName:fff38d09-d990-491f-b780-819f4c390998 nodeName:}" failed. No retries permitted until 2026-07-14 18:55:16.922806877 +0000 UTC m=+2018.460254660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fff38d09-d990-491f-b780-819f4c390998-maas-api-tls") pod "maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" (UID: "fff38d09-d990-491f-b780-819f4c390998") : secret "maas-api-e2e-mt-34c47f4d-serving-cert" not found Jul 14 18:55:16.423322 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.423267 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff38d09-d990-491f-b780-819f4c390998-openshift-service-ca\") pod \"maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g\" (UID: \"fff38d09-d990-491f-b780-819f4c390998\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:16.454517 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.454494 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z5rq\" (UniqueName: \"kubernetes.io/projected/fff38d09-d990-491f-b780-819f4c390998-kube-api-access-5z5rq\") pod \"maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g\" (UID: \"fff38d09-d990-491f-b780-819f4c390998\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:16.910324 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.910269 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:16.927748 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:16.927714 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff38d09-d990-491f-b780-819f4c390998-maas-api-tls\") pod \"maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g\" (UID: \"fff38d09-d990-491f-b780-819f4c390998\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:16.927895 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:55:16.927858 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-serving-cert: secret "maas-api-e2e-mt-34c47f4d-serving-cert" not found Jul 14 18:55:16.927944 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:55:16.927919 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fff38d09-d990-491f-b780-819f4c390998-maas-api-tls podName:fff38d09-d990-491f-b780-819f4c390998 nodeName:}" failed. No retries permitted until 2026-07-14 18:55:17.927904661 +0000 UTC m=+2019.465352429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fff38d09-d990-491f-b780-819f4c390998-maas-api-tls") pod "maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" (UID: "fff38d09-d990-491f-b780-819f4c390998") : secret "maas-api-e2e-mt-34c47f4d-serving-cert" not found Jul 14 18:55:17.011826 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:17.011790 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:17.122650 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:17.122597 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:17.938094 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:17.938059 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff38d09-d990-491f-b780-819f4c390998-maas-api-tls\") pod \"maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g\" (UID: \"fff38d09-d990-491f-b780-819f4c390998\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:17.940660 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:17.940629 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff38d09-d990-491f-b780-819f4c390998-maas-api-tls\") pod \"maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g\" (UID: \"fff38d09-d990-491f-b780-819f4c390998\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:18.063460 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:18.063430 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:18.210570 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:18.210542 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g"] Jul 14 18:55:18.212317 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:55:18.212276 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff38d09_d990_491f_b780_819f4c390998.slice/crio-51e41c59620adb074292a4a5cb268e822c3582336da077a068a65bbed2c6d7eb WatchSource:0}: Error finding container 51e41c59620adb074292a4a5cb268e822c3582336da077a068a65bbed2c6d7eb: Status 404 returned error can't find the container with id 51e41c59620adb074292a4a5cb268e822c3582336da077a068a65bbed2c6d7eb Jul 14 18:55:18.581516 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:18.581422 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" event={"ID":"fff38d09-d990-491f-b780-819f4c390998","Type":"ContainerStarted","Data":"51e41c59620adb074292a4a5cb268e822c3582336da077a068a65bbed2c6d7eb"} Jul 14 18:55:19.588653 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:19.588620 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" event={"ID":"fff38d09-d990-491f-b780-819f4c390998","Type":"ContainerStarted","Data":"4fa098eb6a25bfd53c317934a9370379abea2ce4f4836d2af4e2e97bf4946cdd"} Jul 14 18:55:19.589113 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:19.588726 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:19.610176 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:19.610121 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" podStartSLOduration=3.096114073 podStartE2EDuration="3.610100652s" totalImagesPullingTime="513.986579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:19.607501862 +0000 UTC m=+2021.144949648" watchObservedRunningTime="2026-07-14 18:55:19.610100652 +0000 UTC m=+2021.147548439" Jul 14 18:55:20.714606 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:20.714571 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:21.593221 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:21.593179 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:21.779246 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:21.779210 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:22.071114 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:22.071076 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:25.599126 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:25.599092 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:28.338678 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:28.338641 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:28.438522 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:28.438480 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:35.457387 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:35.457355 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:39.352424 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.352383 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g"] Jul 14 18:55:39.352905 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.352644 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" podUID="fff38d09-d990-491f-b780-819f4c390998" containerName="maas-api" containerID="cri-o://4fa098eb6a25bfd53c317934a9370379abea2ce4f4836d2af4e2e97bf4946cdd" gracePeriod=30 Jul 14 18:55:39.378869 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.378799 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:39.622417 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.622393 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:39.701606 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.701573 2573 generic.go:356] "Generic (PLEG): container finished" podID="fff38d09-d990-491f-b780-819f4c390998" containerID="4fa098eb6a25bfd53c317934a9370379abea2ce4f4836d2af4e2e97bf4946cdd" exitCode=0 Jul 14 18:55:39.701774 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.701621 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" Jul 14 18:55:39.701774 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.701659 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" event={"ID":"fff38d09-d990-491f-b780-819f4c390998","Type":"ContainerDied","Data":"4fa098eb6a25bfd53c317934a9370379abea2ce4f4836d2af4e2e97bf4946cdd"} Jul 14 18:55:39.701774 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.701703 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g" event={"ID":"fff38d09-d990-491f-b780-819f4c390998","Type":"ContainerDied","Data":"51e41c59620adb074292a4a5cb268e822c3582336da077a068a65bbed2c6d7eb"} Jul 14 18:55:39.701774 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.701725 2573 scope.go:117] "RemoveContainer" containerID="4fa098eb6a25bfd53c317934a9370379abea2ce4f4836d2af4e2e97bf4946cdd" Jul 14 18:55:39.713208 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.713193 2573 scope.go:117] "RemoveContainer" containerID="4fa098eb6a25bfd53c317934a9370379abea2ce4f4836d2af4e2e97bf4946cdd" Jul 14 18:55:39.713690 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:55:39.713665 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa098eb6a25bfd53c317934a9370379abea2ce4f4836d2af4e2e97bf4946cdd\": container with ID starting with 4fa098eb6a25bfd53c317934a9370379abea2ce4f4836d2af4e2e97bf4946cdd not found: ID does not exist" containerID="4fa098eb6a25bfd53c317934a9370379abea2ce4f4836d2af4e2e97bf4946cdd" Jul 14 18:55:39.713783 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.713697 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa098eb6a25bfd53c317934a9370379abea2ce4f4836d2af4e2e97bf4946cdd"} err="failed to get container status \"4fa098eb6a25bfd53c317934a9370379abea2ce4f4836d2af4e2e97bf4946cdd\": rpc error: code = NotFound desc = could not find container \"4fa098eb6a25bfd53c317934a9370379abea2ce4f4836d2af4e2e97bf4946cdd\": container with ID starting with 4fa098eb6a25bfd53c317934a9370379abea2ce4f4836d2af4e2e97bf4946cdd not found: ID does not exist" Jul 14 18:55:39.744889 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.744867 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff38d09-d990-491f-b780-819f4c390998-openshift-service-ca\") pod \"fff38d09-d990-491f-b780-819f4c390998\" (UID: \"fff38d09-d990-491f-b780-819f4c390998\") " Jul 14 18:55:39.744992 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.744926 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff38d09-d990-491f-b780-819f4c390998-maas-api-tls\") pod \"fff38d09-d990-491f-b780-819f4c390998\" (UID: \"fff38d09-d990-491f-b780-819f4c390998\") " Jul 14 18:55:39.745038 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.744989 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z5rq\" (UniqueName: \"kubernetes.io/projected/fff38d09-d990-491f-b780-819f4c390998-kube-api-access-5z5rq\") pod \"fff38d09-d990-491f-b780-819f4c390998\" (UID: \"fff38d09-d990-491f-b780-819f4c390998\") " Jul 14 18:55:39.745188 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.745168 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff38d09-d990-491f-b780-819f4c390998-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fff38d09-d990-491f-b780-819f4c390998" (UID: "fff38d09-d990-491f-b780-819f4c390998"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:55:39.745429 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.745400 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff38d09-d990-491f-b780-819f4c390998-openshift-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:55:39.746901 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.746882 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff38d09-d990-491f-b780-819f4c390998-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fff38d09-d990-491f-b780-819f4c390998" (UID: "fff38d09-d990-491f-b780-819f4c390998"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:55:39.746983 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.746954 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff38d09-d990-491f-b780-819f4c390998-kube-api-access-5z5rq" (OuterVolumeSpecName: "kube-api-access-5z5rq") pod "fff38d09-d990-491f-b780-819f4c390998" (UID: "fff38d09-d990-491f-b780-819f4c390998"). InnerVolumeSpecName "kube-api-access-5z5rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:55:39.846604 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.846580 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff38d09-d990-491f-b780-819f4c390998-maas-api-tls\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:55:39.846604 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:39.846605 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z5rq\" (UniqueName: \"kubernetes.io/projected/fff38d09-d990-491f-b780-819f4c390998-kube-api-access-5z5rq\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:55:40.029982 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:40.029947 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g"] Jul 14 18:55:40.033789 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:40.033763 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-34c47f4d-58b48c7695-79c6g"] Jul 14 18:55:41.223321 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:41.223274 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fff38d09-d990-491f-b780-819f4c390998" path="/var/lib/kubelet/pods/fff38d09-d990-491f-b780-819f4c390998/volumes" Jul 14 18:55:43.854055 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:43.854015 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:51.331891 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:51.331852 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:51.432895 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:51.432862 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:54.610500 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:54.610464 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:54.852563 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:54.852532 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:54.989377 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:54.989346 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:55.077440 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:55.077405 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:55:55.231578 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:55:55.231541 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:01.471816 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:01.471714 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:03.042469 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:03.042427 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:03.152690 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:03.152657 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:09.542787 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:09.542750 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:09.647225 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:09.647188 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:09.794025 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:09.793943 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:09.893651 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:09.893621 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:15.523410 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:15.523372 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:16.027465 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:16.027426 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:16.267088 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:16.267050 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:19.476742 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.476702 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:19.575210 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.575176 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:19.676990 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.676955 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft"] Jul 14 18:56:19.677484 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.677458 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff38d09-d990-491f-b780-819f4c390998" containerName="maas-api" Jul 14 18:56:19.677484 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.677476 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff38d09-d990-491f-b780-819f4c390998" containerName="maas-api" Jul 14 18:56:19.677673 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.677612 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fff38d09-d990-491f-b780-819f4c390998" containerName="maas-api" Jul 14 18:56:19.681071 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.681053 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:56:19.685001 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.684984 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-99a89b27-serving-cert\"" Jul 14 18:56:19.694769 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.694744 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft"] Jul 14 18:56:19.723692 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.723645 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:19.843635 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.843554 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bd51ec-df24-463d-9cf5-a97110e4eec7-maas-api-tls\") pod \"maas-api-e2e-mt-99a89b27-668595fddf-54dft\" (UID: \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:56:19.843635 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.843604 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbx2m\" (UniqueName: \"kubernetes.io/projected/f2bd51ec-df24-463d-9cf5-a97110e4eec7-kube-api-access-rbx2m\") pod \"maas-api-e2e-mt-99a89b27-668595fddf-54dft\" (UID: \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:56:19.843820 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.843728 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bd51ec-df24-463d-9cf5-a97110e4eec7-openshift-service-ca\") pod \"maas-api-e2e-mt-99a89b27-668595fddf-54dft\" (UID: \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:56:19.944354 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.944325 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bd51ec-df24-463d-9cf5-a97110e4eec7-openshift-service-ca\") pod \"maas-api-e2e-mt-99a89b27-668595fddf-54dft\" (UID: \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:56:19.944548 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.944364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bd51ec-df24-463d-9cf5-a97110e4eec7-maas-api-tls\") pod \"maas-api-e2e-mt-99a89b27-668595fddf-54dft\" (UID: \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:56:19.944548 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.944395 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbx2m\" (UniqueName: \"kubernetes.io/projected/f2bd51ec-df24-463d-9cf5-a97110e4eec7-kube-api-access-rbx2m\") pod \"maas-api-e2e-mt-99a89b27-668595fddf-54dft\" (UID: \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:56:19.944548 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:56:19.944516 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-serving-cert: secret "maas-api-e2e-mt-99a89b27-serving-cert" not found Jul 14 18:56:19.944729 ip-10-0-132-224 kubenswrapper[2573]: E0714 18:56:19.944606 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2bd51ec-df24-463d-9cf5-a97110e4eec7-maas-api-tls podName:f2bd51ec-df24-463d-9cf5-a97110e4eec7 nodeName:}" failed. No retries permitted until 2026-07-14 18:56:20.444581353 +0000 UTC m=+2081.982029118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f2bd51ec-df24-463d-9cf5-a97110e4eec7-maas-api-tls") pod "maas-api-e2e-mt-99a89b27-668595fddf-54dft" (UID: "f2bd51ec-df24-463d-9cf5-a97110e4eec7") : secret "maas-api-e2e-mt-99a89b27-serving-cert" not found Jul 14 18:56:19.944979 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.944960 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bd51ec-df24-463d-9cf5-a97110e4eec7-openshift-service-ca\") pod \"maas-api-e2e-mt-99a89b27-668595fddf-54dft\" (UID: \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:56:19.955165 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:19.955144 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbx2m\" (UniqueName: \"kubernetes.io/projected/f2bd51ec-df24-463d-9cf5-a97110e4eec7-kube-api-access-rbx2m\") pod \"maas-api-e2e-mt-99a89b27-668595fddf-54dft\" (UID: \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:56:20.222920 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:20.222888 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:20.324553 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:20.324511 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:20.449460 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:20.449410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bd51ec-df24-463d-9cf5-a97110e4eec7-maas-api-tls\") pod \"maas-api-e2e-mt-99a89b27-668595fddf-54dft\" (UID: \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:56:20.451835 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:20.451809 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bd51ec-df24-463d-9cf5-a97110e4eec7-maas-api-tls\") pod \"maas-api-e2e-mt-99a89b27-668595fddf-54dft\" (UID: \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:56:20.592324 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:20.592217 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:56:20.743608 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:20.743578 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft"] Jul 14 18:56:20.744888 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:56:20.744861 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2bd51ec_df24_463d_9cf5_a97110e4eec7.slice/crio-d53ab1a4af4c833ac3cb8064fc19e29e7c5150cfc903601e11c1b74e92f6be50 WatchSource:0}: Error finding container d53ab1a4af4c833ac3cb8064fc19e29e7c5150cfc903601e11c1b74e92f6be50: Status 404 returned error can't find the container with id d53ab1a4af4c833ac3cb8064fc19e29e7c5150cfc903601e11c1b74e92f6be50 Jul 14 18:56:20.746312 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:20.746266 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:56:20.944220 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:20.944180 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" event={"ID":"f2bd51ec-df24-463d-9cf5-a97110e4eec7","Type":"ContainerStarted","Data":"d53ab1a4af4c833ac3cb8064fc19e29e7c5150cfc903601e11c1b74e92f6be50"} Jul 14 18:56:21.950941 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:21.950905 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" event={"ID":"f2bd51ec-df24-463d-9cf5-a97110e4eec7","Type":"ContainerStarted","Data":"c120c0a390ae6b41be19b0ab5412b176e529874b6eb42e7b957f883f1fb601cb"} Jul 14 18:56:21.951393 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:21.950963 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:56:21.974959 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:21.974914 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" podStartSLOduration=2.593368709 podStartE2EDuration="2.974899972s" totalImagesPullingTime="381.531263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:56:21.971443741 +0000 UTC m=+2083.508891526" watchObservedRunningTime="2026-07-14 18:56:21.974899972 +0000 UTC m=+2083.512347757" Jul 14 18:56:24.632753 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:24.632707 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:27.963133 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:27.963106 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:56:33.913547 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:33.913511 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:56:39.049763 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:39.049691 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:56:39.056216 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:39.056191 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 18:56:39.057871 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:39.057846 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:56:39.063332 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:56:39.063312 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 18:57:02.937518 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:02.937475 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft"] Jul 14 18:57:02.938092 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:02.937750 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" podUID="f2bd51ec-df24-463d-9cf5-a97110e4eec7" containerName="maas-api" containerID="cri-o://c120c0a390ae6b41be19b0ab5412b176e529874b6eb42e7b957f883f1fb601cb" gracePeriod=30 Jul 14 18:57:02.957978 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:02.957792 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" podUID="f2bd51ec-df24-463d-9cf5-a97110e4eec7" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.65:8443/health\": dial tcp 10.132.0.65:8443: connect: connection refused" Jul 14 18:57:02.959269 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:02.959235 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:57:03.191969 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:03.191937 2573 generic.go:356] "Generic (PLEG): container finished" podID="f2bd51ec-df24-463d-9cf5-a97110e4eec7" containerID="c120c0a390ae6b41be19b0ab5412b176e529874b6eb42e7b957f883f1fb601cb" exitCode=0 Jul 14 18:57:03.192101 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:03.192003 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" event={"ID":"f2bd51ec-df24-463d-9cf5-a97110e4eec7","Type":"ContainerDied","Data":"c120c0a390ae6b41be19b0ab5412b176e529874b6eb42e7b957f883f1fb601cb"} Jul 14 18:57:03.192101 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:03.192051 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" event={"ID":"f2bd51ec-df24-463d-9cf5-a97110e4eec7","Type":"ContainerDied","Data":"d53ab1a4af4c833ac3cb8064fc19e29e7c5150cfc903601e11c1b74e92f6be50"} Jul 14 18:57:03.192101 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:03.192067 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d53ab1a4af4c833ac3cb8064fc19e29e7c5150cfc903601e11c1b74e92f6be50" Jul 14 18:57:03.205269 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:03.205240 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:57:03.370100 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:03.370073 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bd51ec-df24-463d-9cf5-a97110e4eec7-maas-api-tls\") pod \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\" (UID: \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\") " Jul 14 18:57:03.370316 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:03.370126 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bd51ec-df24-463d-9cf5-a97110e4eec7-openshift-service-ca\") pod \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\" (UID: \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\") " Jul 14 18:57:03.370316 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:03.370159 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbx2m\" (UniqueName: \"kubernetes.io/projected/f2bd51ec-df24-463d-9cf5-a97110e4eec7-kube-api-access-rbx2m\") pod \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\" (UID: \"f2bd51ec-df24-463d-9cf5-a97110e4eec7\") " Jul 14 18:57:03.370613 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:03.370579 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2bd51ec-df24-463d-9cf5-a97110e4eec7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f2bd51ec-df24-463d-9cf5-a97110e4eec7" (UID: "f2bd51ec-df24-463d-9cf5-a97110e4eec7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:57:03.372185 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:03.372154 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2bd51ec-df24-463d-9cf5-a97110e4eec7-kube-api-access-rbx2m" (OuterVolumeSpecName: "kube-api-access-rbx2m") pod "f2bd51ec-df24-463d-9cf5-a97110e4eec7" (UID: "f2bd51ec-df24-463d-9cf5-a97110e4eec7"). InnerVolumeSpecName "kube-api-access-rbx2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:57:03.372185 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:03.372161 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2bd51ec-df24-463d-9cf5-a97110e4eec7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f2bd51ec-df24-463d-9cf5-a97110e4eec7" (UID: "f2bd51ec-df24-463d-9cf5-a97110e4eec7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:57:03.471192 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:03.471157 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bd51ec-df24-463d-9cf5-a97110e4eec7-maas-api-tls\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:57:03.471192 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:03.471193 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bd51ec-df24-463d-9cf5-a97110e4eec7-openshift-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:57:03.471393 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:03.471207 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbx2m\" (UniqueName: \"kubernetes.io/projected/f2bd51ec-df24-463d-9cf5-a97110e4eec7-kube-api-access-rbx2m\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 18:57:04.198027 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:04.197974 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft" Jul 14 18:57:04.221854 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:04.221825 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft"] Jul 14 18:57:04.225692 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:04.225667 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-99a89b27-668595fddf-54dft"] Jul 14 18:57:05.222571 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:05.222534 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2bd51ec-df24-463d-9cf5-a97110e4eec7" path="/var/lib/kubelet/pods/f2bd51ec-df24-463d-9cf5-a97110e4eec7/volumes" Jul 14 18:57:07.569038 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:07.569000 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:57:12.635944 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:12.635902 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:57:36.591236 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:36.591144 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:57:50.345781 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:50.345733 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:57:50.568701 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:50.568663 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:57:50.815103 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:50.815071 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:57:50.887686 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:50.887646 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:57:56.518662 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:56.518626 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:57:56.639464 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:57:56.639431 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:00.096501 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:00.096467 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:00.303661 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:00.303625 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:01.512074 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:01.512038 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:01.611373 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:01.611329 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:01.708983 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:01.708949 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:02.537869 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:02.537810 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:02.635258 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:02.635226 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:02.735976 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:02.735934 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:04.265966 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:04.265934 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:10.801717 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:10.801682 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:10.907991 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:10.907956 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:11.558228 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:11.558192 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:17.941972 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:17.941936 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:18.009460 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:18.009425 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:18.138543 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:18.138508 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:18.238593 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:18.238507 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:34.438446 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:34.438406 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:34.827294 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:34.827260 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:48.847850 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:48.847813 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:50.366132 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:50.366095 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:50.431643 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:50.431607 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:58:55.148687 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:58:55.148658 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:01.311115 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:01.311027 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:01.408659 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:01.408624 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:01.691732 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:01.691704 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:02.017097 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:02.017008 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:02.523891 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:02.523858 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:05.280229 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.280189 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:05.302180 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.302148 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq"] Jul 14 18:59:05.302847 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.302826 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2bd51ec-df24-463d-9cf5-a97110e4eec7" containerName="maas-api" Jul 14 18:59:05.302899 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.302854 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2bd51ec-df24-463d-9cf5-a97110e4eec7" containerName="maas-api" Jul 14 18:59:05.303002 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.302989 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2bd51ec-df24-463d-9cf5-a97110e4eec7" containerName="maas-api" Jul 14 18:59:05.306642 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.306625 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" Jul 14 18:59:05.309528 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.309505 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-0895b5-serving-cert\"" Jul 14 18:59:05.316482 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.316457 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq"] Jul 14 18:59:05.374142 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.374108 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9a9525-d166-4140-9725-acfac04ea571-maas-api-tls\") pod \"maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq\" (UID: \"2a9a9525-d166-4140-9725-acfac04ea571\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" Jul 14 18:59:05.374322 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.374219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qpgs\" (UniqueName: \"kubernetes.io/projected/2a9a9525-d166-4140-9725-acfac04ea571-kube-api-access-2qpgs\") pod \"maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq\" (UID: \"2a9a9525-d166-4140-9725-acfac04ea571\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" Jul 14 18:59:05.374398 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.374342 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9a9525-d166-4140-9725-acfac04ea571-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq\" (UID: \"2a9a9525-d166-4140-9725-acfac04ea571\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" Jul 14 18:59:05.402276 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.402217 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:05.475737 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.475705 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9a9525-d166-4140-9725-acfac04ea571-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq\" (UID: \"2a9a9525-d166-4140-9725-acfac04ea571\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" Jul 14 18:59:05.475949 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.475764 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9a9525-d166-4140-9725-acfac04ea571-maas-api-tls\") pod \"maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq\" (UID: \"2a9a9525-d166-4140-9725-acfac04ea571\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" Jul 14 18:59:05.475949 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.475829 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qpgs\" (UniqueName: \"kubernetes.io/projected/2a9a9525-d166-4140-9725-acfac04ea571-kube-api-access-2qpgs\") pod \"maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq\" (UID: \"2a9a9525-d166-4140-9725-acfac04ea571\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" Jul 14 18:59:05.476386 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.476344 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9a9525-d166-4140-9725-acfac04ea571-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq\" (UID: \"2a9a9525-d166-4140-9725-acfac04ea571\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" Jul 14 18:59:05.478208 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.478188 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9a9525-d166-4140-9725-acfac04ea571-maas-api-tls\") pod \"maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq\" (UID: \"2a9a9525-d166-4140-9725-acfac04ea571\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" Jul 14 18:59:05.498208 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.498176 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qpgs\" (UniqueName: \"kubernetes.io/projected/2a9a9525-d166-4140-9725-acfac04ea571-kube-api-access-2qpgs\") pod \"maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq\" (UID: \"2a9a9525-d166-4140-9725-acfac04ea571\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" Jul 14 18:59:05.601104 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.601016 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:05.617968 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.617943 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" Jul 14 18:59:05.766507 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.766483 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq"] Jul 14 18:59:05.768361 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:59:05.768334 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9a9525_d166_4140_9725_acfac04ea571.slice/crio-fbd090653bd881c003cc462bf1d98bff6eac4f6d8dd3982db70371caa0971719 WatchSource:0}: Error finding container fbd090653bd881c003cc462bf1d98bff6eac4f6d8dd3982db70371caa0971719: Status 404 returned error can't find the container with id fbd090653bd881c003cc462bf1d98bff6eac4f6d8dd3982db70371caa0971719 Jul 14 18:59:05.901270 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.901184 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" event={"ID":"2a9a9525-d166-4140-9725-acfac04ea571","Type":"ContainerStarted","Data":"fbd090653bd881c003cc462bf1d98bff6eac4f6d8dd3982db70371caa0971719"} Jul 14 18:59:05.925060 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:05.925025 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:06.024973 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:06.024942 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:07.913770 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:07.913735 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" event={"ID":"2a9a9525-d166-4140-9725-acfac04ea571","Type":"ContainerStarted","Data":"a4ebef65fe63fd9a352b6e885d272329a738e7f536011fbd9f15d69ff2f3db85"} Jul 14 18:59:07.914178 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:07.913794 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" Jul 14 18:59:07.934948 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:07.934895 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" podStartSLOduration=1.832354093 podStartE2EDuration="2.934878732s" totalImagesPullingTime="1.102524639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:59:07.934179214 +0000 UTC m=+2249.471627002" watchObservedRunningTime="2026-07-14 18:59:07.934878732 +0000 UTC m=+2249.472326516" Jul 14 18:59:09.726108 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:09.726075 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:10.072771 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:10.072665 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:10.398382 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:10.398280 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:10.484540 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:10.484503 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:13.232333 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.232297 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:13.359248 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.359212 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:13.478996 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.478966 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc"] Jul 14 18:59:13.483330 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.483270 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" Jul 14 18:59:13.486181 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.486157 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-08413e-serving-cert\"" Jul 14 18:59:13.492447 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.492425 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc"] Jul 14 18:59:13.533848 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.533810 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:13.549497 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.549469 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d23ad2de-c3a8-4836-a147-719252d9e9d8-maas-api-tls\") pod \"maas-api-e2e-inf-b-08413e-75787d68cd-f82mc\" (UID: \"d23ad2de-c3a8-4836-a147-719252d9e9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" Jul 14 18:59:13.549624 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.549612 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47sl7\" (UniqueName: \"kubernetes.io/projected/d23ad2de-c3a8-4836-a147-719252d9e9d8-kube-api-access-47sl7\") pod \"maas-api-e2e-inf-b-08413e-75787d68cd-f82mc\" (UID: \"d23ad2de-c3a8-4836-a147-719252d9e9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" Jul 14 18:59:13.549698 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.549683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d23ad2de-c3a8-4836-a147-719252d9e9d8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-08413e-75787d68cd-f82mc\" (UID: \"d23ad2de-c3a8-4836-a147-719252d9e9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" Jul 14 18:59:13.634605 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.634565 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:13.651000 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.650973 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d23ad2de-c3a8-4836-a147-719252d9e9d8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-08413e-75787d68cd-f82mc\" (UID: \"d23ad2de-c3a8-4836-a147-719252d9e9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" Jul 14 18:59:13.651119 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.651037 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d23ad2de-c3a8-4836-a147-719252d9e9d8-maas-api-tls\") pod \"maas-api-e2e-inf-b-08413e-75787d68cd-f82mc\" (UID: \"d23ad2de-c3a8-4836-a147-719252d9e9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" Jul 14 18:59:13.651189 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.651117 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47sl7\" (UniqueName: \"kubernetes.io/projected/d23ad2de-c3a8-4836-a147-719252d9e9d8-kube-api-access-47sl7\") pod \"maas-api-e2e-inf-b-08413e-75787d68cd-f82mc\" (UID: \"d23ad2de-c3a8-4836-a147-719252d9e9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" Jul 14 18:59:13.651610 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.651592 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d23ad2de-c3a8-4836-a147-719252d9e9d8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-08413e-75787d68cd-f82mc\" (UID: \"d23ad2de-c3a8-4836-a147-719252d9e9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" Jul 14 18:59:13.653538 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.653520 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d23ad2de-c3a8-4836-a147-719252d9e9d8-maas-api-tls\") pod \"maas-api-e2e-inf-b-08413e-75787d68cd-f82mc\" (UID: \"d23ad2de-c3a8-4836-a147-719252d9e9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" Jul 14 18:59:13.663896 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.663869 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47sl7\" (UniqueName: \"kubernetes.io/projected/d23ad2de-c3a8-4836-a147-719252d9e9d8-kube-api-access-47sl7\") pod \"maas-api-e2e-inf-b-08413e-75787d68cd-f82mc\" (UID: \"d23ad2de-c3a8-4836-a147-719252d9e9d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" Jul 14 18:59:13.795598 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.795523 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" Jul 14 18:59:13.926775 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.926747 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" Jul 14 18:59:13.953824 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:13.953799 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc"] Jul 14 18:59:13.956346 ip-10-0-132-224 kubenswrapper[2573]: W0714 18:59:13.956295 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23ad2de_c3a8_4836_a147_719252d9e9d8.slice/crio-e2e1300eeb18ff1107585fae8954e4acbda4cdfcf5b80e460f27eadd3e2c455b WatchSource:0}: Error finding container e2e1300eeb18ff1107585fae8954e4acbda4cdfcf5b80e460f27eadd3e2c455b: Status 404 returned error can't find the container with id e2e1300eeb18ff1107585fae8954e4acbda4cdfcf5b80e460f27eadd3e2c455b Jul 14 18:59:14.301755 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:14.301720 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:14.394932 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:14.394899 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:14.955890 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:14.955854 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" event={"ID":"d23ad2de-c3a8-4836-a147-719252d9e9d8","Type":"ContainerStarted","Data":"e954093ca3cf64d9acb28e4b20462594a58f91da0003ede7a2296a4ddb2ef15a"} Jul 14 18:59:14.956069 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:14.955904 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" event={"ID":"d23ad2de-c3a8-4836-a147-719252d9e9d8","Type":"ContainerStarted","Data":"e2e1300eeb18ff1107585fae8954e4acbda4cdfcf5b80e460f27eadd3e2c455b"} Jul 14 18:59:14.956069 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:14.955936 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" Jul 14 18:59:14.976363 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:14.976319 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" podStartSLOduration=1.6186122250000001 podStartE2EDuration="1.976301535s" totalImagesPullingTime="357.68931ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:59:14.974222865 +0000 UTC m=+2256.511670651" watchObservedRunningTime="2026-07-14 18:59:14.976301535 +0000 UTC m=+2256.513749317" Jul 14 18:59:17.794468 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:17.794428 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:20.966325 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:20.966273 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" Jul 14 18:59:23.499520 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:23.499482 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:26.671071 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:26.671029 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:26.884832 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:26.884791 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:36.286950 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:36.286910 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:36.387604 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:36.387568 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 18:59:44.504920 ip-10-0-132-224 kubenswrapper[2573]: I0714 18:59:44.504885 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:00.140258 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.140214 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734260-kq6xr"] Jul 14 19:00:00.144194 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.144168 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0895b5-29734260-ktdmn"] Jul 14 19:00:00.144355 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.144331 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734260-kq6xr" Jul 14 19:00:00.147013 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.146991 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 19:00:00.147762 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.147746 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-08413e-29734260-m4nph"] Jul 14 19:00:00.147885 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.147869 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0895b5-29734260-ktdmn" Jul 14 19:00:00.152022 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.151996 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734260-kq6xr"] Jul 14 19:00:00.152161 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.152099 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-08413e-29734260-m4nph" Jul 14 19:00:00.155552 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.155501 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-08413e-29734260-m4nph"] Jul 14 19:00:00.158860 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.158836 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0895b5-29734260-ktdmn"] Jul 14 19:00:00.159634 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.159523 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734260-kq6xr" Jul 14 19:00:00.169988 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.169963 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0895b5-29734260-ktdmn" Jul 14 19:00:00.205278 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.205252 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-08413e-29734260-m4nph" Jul 14 19:00:00.393570 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.393545 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-08413e-29734260-m4nph"] Jul 14 19:00:00.395398 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:00:00.395371 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2343a749_132f_49a3_b90a_c75566a6e093.slice/crio-453b5b0430df130dbb90e959073e543c2d465688d2540c41380c52e1e69c2e37 WatchSource:0}: Error finding container 453b5b0430df130dbb90e959073e543c2d465688d2540c41380c52e1e69c2e37: Status 404 returned error can't find the container with id 453b5b0430df130dbb90e959073e543c2d465688d2540c41380c52e1e69c2e37 Jul 14 19:00:00.569924 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.569901 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0895b5-29734260-ktdmn"] Jul 14 19:00:00.572863 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:00:00.572831 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947f96a7_0834_494c_ab3d_7789507f5d05.slice/crio-87009ff0ec117b9b3a3cd0a7b3f02c13d302a33757be9e431bac3766378a3775 WatchSource:0}: Error finding container 87009ff0ec117b9b3a3cd0a7b3f02c13d302a33757be9e431bac3766378a3775: Status 404 returned error can't find the container with id 87009ff0ec117b9b3a3cd0a7b3f02c13d302a33757be9e431bac3766378a3775 Jul 14 19:00:00.576625 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:00.576603 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734260-kq6xr"] Jul 14 19:00:00.577028 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:00:00.577007 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd212eb6_7b43_4b30_85c8_57d1dfe4078c.slice/crio-3d9efa040e9ba2b9ff0d7094127f2be9c5d505fb1ba013b808675e5a21185f76 WatchSource:0}: Error finding container 3d9efa040e9ba2b9ff0d7094127f2be9c5d505fb1ba013b808675e5a21185f76: Status 404 returned error can't find the container with id 3d9efa040e9ba2b9ff0d7094127f2be9c5d505fb1ba013b808675e5a21185f76 Jul 14 19:00:01.224596 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:01.224562 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0895b5-29734260-ktdmn" event={"ID":"947f96a7-0834-494c-ab3d-7789507f5d05","Type":"ContainerStarted","Data":"3d73ced0d5690541e9bac1dc37cdb1dc04dbe64b35c398cecf277f86d89340f2"} Jul 14 19:00:01.225024 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:01.224624 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0895b5-29734260-ktdmn" event={"ID":"947f96a7-0834-494c-ab3d-7789507f5d05","Type":"ContainerStarted","Data":"87009ff0ec117b9b3a3cd0a7b3f02c13d302a33757be9e431bac3766378a3775"} Jul 14 19:00:01.226096 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:01.226073 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-08413e-29734260-m4nph" event={"ID":"2343a749-132f-49a3-b90a-c75566a6e093","Type":"ContainerStarted","Data":"ccaadf164738bc7f1f7786ad6c8b35a5b72bc73b7b729be242d64d10edb87227"} Jul 14 19:00:01.226187 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:01.226115 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-08413e-29734260-m4nph" event={"ID":"2343a749-132f-49a3-b90a-c75566a6e093","Type":"ContainerStarted","Data":"453b5b0430df130dbb90e959073e543c2d465688d2540c41380c52e1e69c2e37"} Jul 14 19:00:01.227481 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:01.227457 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734260-kq6xr" event={"ID":"fd212eb6-7b43-4b30-85c8-57d1dfe4078c","Type":"ContainerStarted","Data":"3c4e165c1191ce686030e8fe64d8083a2875e3bdbf31c42cb9644c6eefa8167f"} Jul 14 19:00:01.227592 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:01.227502 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734260-kq6xr" event={"ID":"fd212eb6-7b43-4b30-85c8-57d1dfe4078c","Type":"ContainerStarted","Data":"3d9efa040e9ba2b9ff0d7094127f2be9c5d505fb1ba013b808675e5a21185f76"} Jul 14 19:00:01.244556 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:01.244512 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0895b5-29734260-ktdmn" podStartSLOduration=1.244498896 podStartE2EDuration="1.244498896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:00:01.243379078 +0000 UTC m=+2302.780826860" watchObservedRunningTime="2026-07-14 19:00:01.244498896 +0000 UTC m=+2302.781946681" Jul 14 19:00:01.260437 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:01.260396 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734260-kq6xr" podStartSLOduration=1.260382424 podStartE2EDuration="1.260382424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:00:01.258054795 +0000 UTC m=+2302.795502581" watchObservedRunningTime="2026-07-14 19:00:01.260382424 +0000 UTC m=+2302.797830209" Jul 14 19:00:01.274033 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:01.273956 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-08413e-29734260-m4nph" podStartSLOduration=1.273943239 podStartE2EDuration="1.273943239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:00:01.273402619 +0000 UTC m=+2302.810850396" watchObservedRunningTime="2026-07-14 19:00:01.273943239 +0000 UTC m=+2302.811391023" Jul 14 19:00:02.235572 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:02.235541 2573 generic.go:356] "Generic (PLEG): container finished" podID="2343a749-132f-49a3-b90a-c75566a6e093" containerID="ccaadf164738bc7f1f7786ad6c8b35a5b72bc73b7b729be242d64d10edb87227" exitCode=0 Jul 14 19:00:02.236013 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:02.235639 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-08413e-29734260-m4nph" event={"ID":"2343a749-132f-49a3-b90a-c75566a6e093","Type":"ContainerDied","Data":"ccaadf164738bc7f1f7786ad6c8b35a5b72bc73b7b729be242d64d10edb87227"} Jul 14 19:00:02.237090 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:02.237068 2573 generic.go:356] "Generic (PLEG): container finished" podID="fd212eb6-7b43-4b30-85c8-57d1dfe4078c" containerID="3c4e165c1191ce686030e8fe64d8083a2875e3bdbf31c42cb9644c6eefa8167f" exitCode=0 Jul 14 19:00:02.237185 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:02.237101 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734260-kq6xr" event={"ID":"fd212eb6-7b43-4b30-85c8-57d1dfe4078c","Type":"ContainerDied","Data":"3c4e165c1191ce686030e8fe64d8083a2875e3bdbf31c42cb9644c6eefa8167f"} Jul 14 19:00:02.238622 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:02.238598 2573 generic.go:356] "Generic (PLEG): container finished" podID="947f96a7-0834-494c-ab3d-7789507f5d05" containerID="3d73ced0d5690541e9bac1dc37cdb1dc04dbe64b35c398cecf277f86d89340f2" exitCode=0 Jul 14 19:00:02.238725 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:02.238626 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0895b5-29734260-ktdmn" event={"ID":"947f96a7-0834-494c-ab3d-7789507f5d05","Type":"ContainerDied","Data":"3d73ced0d5690541e9bac1dc37cdb1dc04dbe64b35c398cecf277f86d89340f2"} Jul 14 19:00:03.405090 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:03.405064 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0895b5-29734260-ktdmn" Jul 14 19:00:03.440906 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:03.440876 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-08413e-29734260-m4nph" Jul 14 19:00:03.462263 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:03.462239 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734260-kq6xr" Jul 14 19:00:04.251269 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:04.251239 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0895b5-29734260-ktdmn" Jul 14 19:00:04.251473 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:04.251311 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0895b5-29734260-ktdmn" event={"ID":"947f96a7-0834-494c-ab3d-7789507f5d05","Type":"ContainerDied","Data":"87009ff0ec117b9b3a3cd0a7b3f02c13d302a33757be9e431bac3766378a3775"} Jul 14 19:00:04.251473 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:04.251357 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87009ff0ec117b9b3a3cd0a7b3f02c13d302a33757be9e431bac3766378a3775" Jul 14 19:00:04.252785 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:04.252762 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-08413e-29734260-m4nph" Jul 14 19:00:04.252785 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:04.252784 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-08413e-29734260-m4nph" event={"ID":"2343a749-132f-49a3-b90a-c75566a6e093","Type":"ContainerDied","Data":"453b5b0430df130dbb90e959073e543c2d465688d2540c41380c52e1e69c2e37"} Jul 14 19:00:04.252977 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:04.252819 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="453b5b0430df130dbb90e959073e543c2d465688d2540c41380c52e1e69c2e37" Jul 14 19:00:04.254143 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:04.254118 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734260-kq6xr" Jul 14 19:00:04.254258 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:04.254158 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734260-kq6xr" event={"ID":"fd212eb6-7b43-4b30-85c8-57d1dfe4078c","Type":"ContainerDied","Data":"3d9efa040e9ba2b9ff0d7094127f2be9c5d505fb1ba013b808675e5a21185f76"} Jul 14 19:00:04.254258 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:04.254196 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d9efa040e9ba2b9ff0d7094127f2be9c5d505fb1ba013b808675e5a21185f76" Jul 14 19:00:06.389274 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:06.389246 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:07.316086 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.316052 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:07.447970 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.447934 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j"] Jul 14 19:00:07.448753 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.448732 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2343a749-132f-49a3-b90a-c75566a6e093" containerName="cleanup" Jul 14 19:00:07.448795 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.448758 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2343a749-132f-49a3-b90a-c75566a6e093" containerName="cleanup" Jul 14 19:00:07.448795 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.448776 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd212eb6-7b43-4b30-85c8-57d1dfe4078c" containerName="cleanup" Jul 14 19:00:07.448795 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.448784 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd212eb6-7b43-4b30-85c8-57d1dfe4078c" containerName="cleanup" Jul 14 19:00:07.448893 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.448817 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="947f96a7-0834-494c-ab3d-7789507f5d05" containerName="cleanup" Jul 14 19:00:07.448893 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.448826 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="947f96a7-0834-494c-ab3d-7789507f5d05" containerName="cleanup" Jul 14 19:00:07.448973 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.448942 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="947f96a7-0834-494c-ab3d-7789507f5d05" containerName="cleanup" Jul 14 19:00:07.448973 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.448954 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd212eb6-7b43-4b30-85c8-57d1dfe4078c" containerName="cleanup" Jul 14 19:00:07.449039 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.448972 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2343a749-132f-49a3-b90a-c75566a6e093" containerName="cleanup" Jul 14 19:00:07.452949 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.452932 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j" Jul 14 19:00:07.456617 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.456595 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 19:00:07.458122 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.458097 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j"] Jul 14 19:00:07.559986 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.559939 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff9bca6-c23e-40ef-840f-14b668eb79a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j\" (UID: \"1ff9bca6-c23e-40ef-840f-14b668eb79a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j" Jul 14 19:00:07.660824 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.660735 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff9bca6-c23e-40ef-840f-14b668eb79a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j\" (UID: \"1ff9bca6-c23e-40ef-840f-14b668eb79a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j" Jul 14 19:00:07.661367 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.661345 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff9bca6-c23e-40ef-840f-14b668eb79a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j\" (UID: \"1ff9bca6-c23e-40ef-840f-14b668eb79a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j" Jul 14 19:00:07.767519 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.767486 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j" Jul 14 19:00:07.909849 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:07.909823 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j"] Jul 14 19:00:07.912067 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:00:07.912012 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff9bca6_c23e_40ef_840f_14b668eb79a9.slice/crio-624a5579865fb82ddbd28f7c271ccd57c7f841d4d350889d799ab74dce861c17 WatchSource:0}: Error finding container 624a5579865fb82ddbd28f7c271ccd57c7f841d4d350889d799ab74dce861c17: Status 404 returned error can't find the container with id 624a5579865fb82ddbd28f7c271ccd57c7f841d4d350889d799ab74dce861c17 Jul 14 19:00:08.282871 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:08.282836 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j" event={"ID":"1ff9bca6-c23e-40ef-840f-14b668eb79a9","Type":"ContainerStarted","Data":"409a3a037b4a0156b1a475c8388cf8fbbfa8966d72ee447340ab5524912daa61"} Jul 14 19:00:08.283040 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:08.282890 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j" event={"ID":"1ff9bca6-c23e-40ef-840f-14b668eb79a9","Type":"ContainerStarted","Data":"624a5579865fb82ddbd28f7c271ccd57c7f841d4d350889d799ab74dce861c17"} Jul 14 19:00:08.300690 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:08.300629 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j" podStartSLOduration=1.300609428 podStartE2EDuration="1.300609428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:00:08.300371938 +0000 UTC m=+2309.837819723" watchObservedRunningTime="2026-07-14 19:00:08.300609428 +0000 UTC m=+2309.838057221" Jul 14 19:00:09.250904 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:09.250862 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:09.290871 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:09.290846 2573 generic.go:356] "Generic (PLEG): container finished" podID="1ff9bca6-c23e-40ef-840f-14b668eb79a9" containerID="409a3a037b4a0156b1a475c8388cf8fbbfa8966d72ee447340ab5524912daa61" exitCode=0 Jul 14 19:00:09.291003 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:09.290877 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j" event={"ID":"1ff9bca6-c23e-40ef-840f-14b668eb79a9","Type":"ContainerDied","Data":"409a3a037b4a0156b1a475c8388cf8fbbfa8966d72ee447340ab5524912daa61"} Jul 14 19:00:09.374671 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:09.374644 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:10.443924 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:10.443898 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j" Jul 14 19:00:10.589706 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:10.589622 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff9bca6-c23e-40ef-840f-14b668eb79a9-maas-api-service-ca\") pod \"1ff9bca6-c23e-40ef-840f-14b668eb79a9\" (UID: \"1ff9bca6-c23e-40ef-840f-14b668eb79a9\") " Jul 14 19:00:10.589988 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:10.589963 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff9bca6-c23e-40ef-840f-14b668eb79a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ff9bca6-c23e-40ef-840f-14b668eb79a9" (UID: "1ff9bca6-c23e-40ef-840f-14b668eb79a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:00:10.690858 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:10.690828 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ff9bca6-c23e-40ef-840f-14b668eb79a9-maas-api-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:00:11.311991 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:11.311957 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j" event={"ID":"1ff9bca6-c23e-40ef-840f-14b668eb79a9","Type":"ContainerDied","Data":"624a5579865fb82ddbd28f7c271ccd57c7f841d4d350889d799ab74dce861c17"} Jul 14 19:00:11.312156 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:11.311993 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j" Jul 14 19:00:11.312156 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:11.312029 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="624a5579865fb82ddbd28f7c271ccd57c7f841d4d350889d799ab74dce861c17" Jul 14 19:00:17.970505 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:17.970466 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq"] Jul 14 19:00:17.973361 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:17.970773 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" podUID="2a9a9525-d166-4140-9725-acfac04ea571" containerName="maas-api" containerID="cri-o://a4ebef65fe63fd9a352b6e885d272329a738e7f536011fbd9f15d69ff2f3db85" gracePeriod=30 Jul 14 19:00:17.999502 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:17.999475 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:18.000266 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.000219 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0895b5-29734260-ktdmn"] Jul 14 19:00:18.003700 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.003680 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0895b5-29734260-ktdmn"] Jul 14 19:00:18.225262 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.225200 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" Jul 14 19:00:18.363330 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.363296 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9a9525-d166-4140-9725-acfac04ea571-maas-api-tls\") pod \"2a9a9525-d166-4140-9725-acfac04ea571\" (UID: \"2a9a9525-d166-4140-9725-acfac04ea571\") " Jul 14 19:00:18.363512 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.363346 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9a9525-d166-4140-9725-acfac04ea571-openshift-service-ca\") pod \"2a9a9525-d166-4140-9725-acfac04ea571\" (UID: \"2a9a9525-d166-4140-9725-acfac04ea571\") " Jul 14 19:00:18.363512 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.363500 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qpgs\" (UniqueName: \"kubernetes.io/projected/2a9a9525-d166-4140-9725-acfac04ea571-kube-api-access-2qpgs\") pod \"2a9a9525-d166-4140-9725-acfac04ea571\" (UID: \"2a9a9525-d166-4140-9725-acfac04ea571\") " Jul 14 19:00:18.363804 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.363775 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9a9525-d166-4140-9725-acfac04ea571-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2a9a9525-d166-4140-9725-acfac04ea571" (UID: "2a9a9525-d166-4140-9725-acfac04ea571"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:00:18.363987 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.363965 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9a9525-d166-4140-9725-acfac04ea571-openshift-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:00:18.364557 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.364533 2573 generic.go:356] "Generic (PLEG): container finished" podID="2a9a9525-d166-4140-9725-acfac04ea571" containerID="a4ebef65fe63fd9a352b6e885d272329a738e7f536011fbd9f15d69ff2f3db85" exitCode=0 Jul 14 19:00:18.364675 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.364575 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" Jul 14 19:00:18.364675 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.364655 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" event={"ID":"2a9a9525-d166-4140-9725-acfac04ea571","Type":"ContainerDied","Data":"a4ebef65fe63fd9a352b6e885d272329a738e7f536011fbd9f15d69ff2f3db85"} Jul 14 19:00:18.364768 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.364703 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq" event={"ID":"2a9a9525-d166-4140-9725-acfac04ea571","Type":"ContainerDied","Data":"fbd090653bd881c003cc462bf1d98bff6eac4f6d8dd3982db70371caa0971719"} Jul 14 19:00:18.364768 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.364728 2573 scope.go:117] "RemoveContainer" containerID="a4ebef65fe63fd9a352b6e885d272329a738e7f536011fbd9f15d69ff2f3db85" Jul 14 19:00:18.365416 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.365391 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a9a9525-d166-4140-9725-acfac04ea571-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2a9a9525-d166-4140-9725-acfac04ea571" (UID: "2a9a9525-d166-4140-9725-acfac04ea571"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:00:18.365759 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.365731 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9a9525-d166-4140-9725-acfac04ea571-kube-api-access-2qpgs" (OuterVolumeSpecName: "kube-api-access-2qpgs") pod "2a9a9525-d166-4140-9725-acfac04ea571" (UID: "2a9a9525-d166-4140-9725-acfac04ea571"). InnerVolumeSpecName "kube-api-access-2qpgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:00:18.386710 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.386694 2573 scope.go:117] "RemoveContainer" containerID="a4ebef65fe63fd9a352b6e885d272329a738e7f536011fbd9f15d69ff2f3db85" Jul 14 19:00:18.387333 ip-10-0-132-224 kubenswrapper[2573]: E0714 19:00:18.387258 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4ebef65fe63fd9a352b6e885d272329a738e7f536011fbd9f15d69ff2f3db85\": container with ID starting with a4ebef65fe63fd9a352b6e885d272329a738e7f536011fbd9f15d69ff2f3db85 not found: ID does not exist" containerID="a4ebef65fe63fd9a352b6e885d272329a738e7f536011fbd9f15d69ff2f3db85" Jul 14 19:00:18.387333 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.387309 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ebef65fe63fd9a352b6e885d272329a738e7f536011fbd9f15d69ff2f3db85"} err="failed to get container status \"a4ebef65fe63fd9a352b6e885d272329a738e7f536011fbd9f15d69ff2f3db85\": rpc error: code = NotFound desc = could not find container \"a4ebef65fe63fd9a352b6e885d272329a738e7f536011fbd9f15d69ff2f3db85\": container with ID starting with a4ebef65fe63fd9a352b6e885d272329a738e7f536011fbd9f15d69ff2f3db85 not found: ID does not exist" Jul 14 19:00:18.465133 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.465102 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qpgs\" (UniqueName: \"kubernetes.io/projected/2a9a9525-d166-4140-9725-acfac04ea571-kube-api-access-2qpgs\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:00:18.465133 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.465129 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a9a9525-d166-4140-9725-acfac04ea571-maas-api-tls\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:00:18.691894 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.691860 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq"] Jul 14 19:00:18.696957 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:18.696932 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0895b5-6888f759cc-9mfdq"] Jul 14 19:00:19.223169 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:19.223139 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a9a9525-d166-4140-9725-acfac04ea571" path="/var/lib/kubelet/pods/2a9a9525-d166-4140-9725-acfac04ea571/volumes" Jul 14 19:00:19.223567 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:19.223470 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="947f96a7-0834-494c-ab3d-7789507f5d05" path="/var/lib/kubelet/pods/947f96a7-0834-494c-ab3d-7789507f5d05/volumes" Jul 14 19:00:22.552565 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:22.552530 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:29.219634 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:29.219593 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:29.292314 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:29.292254 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:33.345361 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:33.345263 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:33.604181 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:33.604084 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:33.749164 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:33.749127 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk"] Jul 14 19:00:33.749641 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:33.749627 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a9a9525-d166-4140-9725-acfac04ea571" containerName="maas-api" Jul 14 19:00:33.749691 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:33.749642 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a9a9525-d166-4140-9725-acfac04ea571" containerName="maas-api" Jul 14 19:00:33.749691 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:33.749663 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff9bca6-c23e-40ef-840f-14b668eb79a9" containerName="revoke-keys" Jul 14 19:00:33.749691 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:33.749669 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff9bca6-c23e-40ef-840f-14b668eb79a9" containerName="revoke-keys" Jul 14 19:00:33.749790 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:33.749757 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a9a9525-d166-4140-9725-acfac04ea571" containerName="maas-api" Jul 14 19:00:33.749790 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:33.749765 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ff9bca6-c23e-40ef-840f-14b668eb79a9" containerName="revoke-keys" Jul 14 19:00:33.754235 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:33.754213 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk" Jul 14 19:00:33.758528 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:33.758509 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 19:00:33.771173 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:33.771142 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk"] Jul 14 19:00:33.811378 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:33.811349 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb6bafe-ed56-435a-9096-6a297379f301-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk\" (UID: \"abb6bafe-ed56-435a-9096-6a297379f301\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk" Jul 14 19:00:33.912610 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:33.912520 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb6bafe-ed56-435a-9096-6a297379f301-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk\" (UID: \"abb6bafe-ed56-435a-9096-6a297379f301\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk" Jul 14 19:00:33.913156 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:33.913136 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb6bafe-ed56-435a-9096-6a297379f301-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk\" (UID: \"abb6bafe-ed56-435a-9096-6a297379f301\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk" Jul 14 19:00:34.064911 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:34.064874 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk" Jul 14 19:00:34.208226 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:34.208199 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk"] Jul 14 19:00:34.210677 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:00:34.210652 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb6bafe_ed56_435a_9096_6a297379f301.slice/crio-c11dd1a6c6c84ab75c512e57e6c4a063a8f1167d7be923d146bf06078f322df0 WatchSource:0}: Error finding container c11dd1a6c6c84ab75c512e57e6c4a063a8f1167d7be923d146bf06078f322df0: Status 404 returned error can't find the container with id c11dd1a6c6c84ab75c512e57e6c4a063a8f1167d7be923d146bf06078f322df0 Jul 14 19:00:34.310623 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:34.310586 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:34.408669 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:34.408629 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:34.461855 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:34.461765 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk" event={"ID":"abb6bafe-ed56-435a-9096-6a297379f301","Type":"ContainerStarted","Data":"c11dd1a6c6c84ab75c512e57e6c4a063a8f1167d7be923d146bf06078f322df0"} Jul 14 19:00:35.469693 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:35.469663 2573 generic.go:356] "Generic (PLEG): container finished" podID="abb6bafe-ed56-435a-9096-6a297379f301" containerID="157893baab8b109f1004f8c96310d470df695da3625b694b30e3a019cd0810ea" exitCode=0 Jul 14 19:00:35.470071 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:35.469706 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk" event={"ID":"abb6bafe-ed56-435a-9096-6a297379f301","Type":"ContainerDied","Data":"157893baab8b109f1004f8c96310d470df695da3625b694b30e3a019cd0810ea"} Jul 14 19:00:36.623307 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:36.623268 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk" Jul 14 19:00:36.739605 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:36.739528 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb6bafe-ed56-435a-9096-6a297379f301-maas-api-service-ca\") pod \"abb6bafe-ed56-435a-9096-6a297379f301\" (UID: \"abb6bafe-ed56-435a-9096-6a297379f301\") " Jul 14 19:00:36.739861 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:36.739839 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb6bafe-ed56-435a-9096-6a297379f301-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abb6bafe-ed56-435a-9096-6a297379f301" (UID: "abb6bafe-ed56-435a-9096-6a297379f301"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:00:36.840998 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:36.840972 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb6bafe-ed56-435a-9096-6a297379f301-maas-api-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:00:37.483233 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:37.483125 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk" event={"ID":"abb6bafe-ed56-435a-9096-6a297379f301","Type":"ContainerDied","Data":"c11dd1a6c6c84ab75c512e57e6c4a063a8f1167d7be923d146bf06078f322df0"} Jul 14 19:00:37.483233 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:37.483162 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk" Jul 14 19:00:37.483233 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:37.483174 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c11dd1a6c6c84ab75c512e57e6c4a063a8f1167d7be923d146bf06078f322df0" Jul 14 19:00:44.332716 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.332682 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc"] Jul 14 19:00:44.333131 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.332955 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" podUID="d23ad2de-c3a8-4836-a147-719252d9e9d8" containerName="maas-api" containerID="cri-o://e954093ca3cf64d9acb28e4b20462594a58f91da0003ede7a2296a4ddb2ef15a" gracePeriod=30 Jul 14 19:00:44.354499 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.354475 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-08413e-29734260-m4nph"] Jul 14 19:00:44.356716 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.356691 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:44.366961 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.366935 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-08413e-29734260-m4nph"] Jul 14 19:00:44.525943 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.525916 2573 generic.go:356] "Generic (PLEG): container finished" podID="d23ad2de-c3a8-4836-a147-719252d9e9d8" containerID="e954093ca3cf64d9acb28e4b20462594a58f91da0003ede7a2296a4ddb2ef15a" exitCode=0 Jul 14 19:00:44.526084 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.525987 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" event={"ID":"d23ad2de-c3a8-4836-a147-719252d9e9d8","Type":"ContainerDied","Data":"e954093ca3cf64d9acb28e4b20462594a58f91da0003ede7a2296a4ddb2ef15a"} Jul 14 19:00:44.594746 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.594690 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" Jul 14 19:00:44.714863 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.714830 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d23ad2de-c3a8-4836-a147-719252d9e9d8-maas-api-tls\") pod \"d23ad2de-c3a8-4836-a147-719252d9e9d8\" (UID: \"d23ad2de-c3a8-4836-a147-719252d9e9d8\") " Jul 14 19:00:44.715004 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.714922 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47sl7\" (UniqueName: \"kubernetes.io/projected/d23ad2de-c3a8-4836-a147-719252d9e9d8-kube-api-access-47sl7\") pod \"d23ad2de-c3a8-4836-a147-719252d9e9d8\" (UID: \"d23ad2de-c3a8-4836-a147-719252d9e9d8\") " Jul 14 19:00:44.715004 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.714953 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d23ad2de-c3a8-4836-a147-719252d9e9d8-openshift-service-ca\") pod \"d23ad2de-c3a8-4836-a147-719252d9e9d8\" (UID: \"d23ad2de-c3a8-4836-a147-719252d9e9d8\") " Jul 14 19:00:44.715364 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.715343 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d23ad2de-c3a8-4836-a147-719252d9e9d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d23ad2de-c3a8-4836-a147-719252d9e9d8" (UID: "d23ad2de-c3a8-4836-a147-719252d9e9d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:00:44.716957 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.716935 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23ad2de-c3a8-4836-a147-719252d9e9d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d23ad2de-c3a8-4836-a147-719252d9e9d8" (UID: "d23ad2de-c3a8-4836-a147-719252d9e9d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:00:44.717052 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.717008 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23ad2de-c3a8-4836-a147-719252d9e9d8-kube-api-access-47sl7" (OuterVolumeSpecName: "kube-api-access-47sl7") pod "d23ad2de-c3a8-4836-a147-719252d9e9d8" (UID: "d23ad2de-c3a8-4836-a147-719252d9e9d8"). InnerVolumeSpecName "kube-api-access-47sl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:00:44.815655 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.815628 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d23ad2de-c3a8-4836-a147-719252d9e9d8-maas-api-tls\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:00:44.815655 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.815650 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47sl7\" (UniqueName: \"kubernetes.io/projected/d23ad2de-c3a8-4836-a147-719252d9e9d8-kube-api-access-47sl7\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:00:44.815655 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:44.815659 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d23ad2de-c3a8-4836-a147-719252d9e9d8-openshift-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:00:45.223836 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:45.223807 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2343a749-132f-49a3-b90a-c75566a6e093" path="/var/lib/kubelet/pods/2343a749-132f-49a3-b90a-c75566a6e093/volumes" Jul 14 19:00:45.533501 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:45.533420 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" Jul 14 19:00:45.533887 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:45.533419 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc" event={"ID":"d23ad2de-c3a8-4836-a147-719252d9e9d8","Type":"ContainerDied","Data":"e2e1300eeb18ff1107585fae8954e4acbda4cdfcf5b80e460f27eadd3e2c455b"} Jul 14 19:00:45.533887 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:45.533556 2573 scope.go:117] "RemoveContainer" containerID="e954093ca3cf64d9acb28e4b20462594a58f91da0003ede7a2296a4ddb2ef15a" Jul 14 19:00:45.552103 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:45.552077 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc"] Jul 14 19:00:45.558353 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:45.558334 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-08413e-75787d68cd-f82mc"] Jul 14 19:00:47.222906 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:47.222863 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d23ad2de-c3a8-4836-a147-719252d9e9d8" path="/var/lib/kubelet/pods/d23ad2de-c3a8-4836-a147-719252d9e9d8/volumes" Jul 14 19:00:48.825328 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:48.825274 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:55.607610 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:55.607572 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:55.743052 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:55.743017 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:59.587406 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:59.587353 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:00:59.841696 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:00:59.841601 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:00.220200 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:00.220157 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:00.569191 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:00.569093 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:00.811665 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:00.811615 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:03.773246 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:03.773204 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:03.907351 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:03.907312 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct"] Jul 14 19:01:03.907801 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:03.907787 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abb6bafe-ed56-435a-9096-6a297379f301" containerName="revoke-keys" Jul 14 19:01:03.907846 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:03.907802 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="abb6bafe-ed56-435a-9096-6a297379f301" containerName="revoke-keys" Jul 14 19:01:03.907846 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:03.907830 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23ad2de-c3a8-4836-a147-719252d9e9d8" containerName="maas-api" Jul 14 19:01:03.907846 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:03.907836 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23ad2de-c3a8-4836-a147-719252d9e9d8" containerName="maas-api" Jul 14 19:01:03.907942 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:03.907931 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="abb6bafe-ed56-435a-9096-6a297379f301" containerName="revoke-keys" Jul 14 19:01:03.907942 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:03.907940 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23ad2de-c3a8-4836-a147-719252d9e9d8" containerName="maas-api" Jul 14 19:01:03.912534 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:03.912518 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" Jul 14 19:01:03.915531 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:03.915510 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-82771c-serving-cert\"" Jul 14 19:01:03.922934 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:03.922909 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct"] Jul 14 19:01:03.948029 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:03.948003 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:04.005472 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:04.005446 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-597qd\" (UniqueName: \"kubernetes.io/projected/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-kube-api-access-597qd\") pod \"maas-api-e2e-api-a-82771c-7d67c95767-59fct\" (UID: \"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" Jul 14 19:01:04.005604 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:04.005512 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-maas-api-tls\") pod \"maas-api-e2e-api-a-82771c-7d67c95767-59fct\" (UID: \"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" Jul 14 19:01:04.005604 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:04.005564 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-openshift-service-ca\") pod \"maas-api-e2e-api-a-82771c-7d67c95767-59fct\" (UID: \"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" Jul 14 19:01:04.106279 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:04.106201 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-597qd\" (UniqueName: \"kubernetes.io/projected/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-kube-api-access-597qd\") pod \"maas-api-e2e-api-a-82771c-7d67c95767-59fct\" (UID: \"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" Jul 14 19:01:04.106449 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:04.106306 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-maas-api-tls\") pod \"maas-api-e2e-api-a-82771c-7d67c95767-59fct\" (UID: \"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" Jul 14 19:01:04.106449 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:04.106398 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-openshift-service-ca\") pod \"maas-api-e2e-api-a-82771c-7d67c95767-59fct\" (UID: \"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" Jul 14 19:01:04.107050 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:04.107026 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-openshift-service-ca\") pod \"maas-api-e2e-api-a-82771c-7d67c95767-59fct\" (UID: \"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" Jul 14 19:01:04.108785 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:04.108765 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-maas-api-tls\") pod \"maas-api-e2e-api-a-82771c-7d67c95767-59fct\" (UID: \"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" Jul 14 19:01:04.116149 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:04.116123 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-597qd\" (UniqueName: \"kubernetes.io/projected/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-kube-api-access-597qd\") pod \"maas-api-e2e-api-a-82771c-7d67c95767-59fct\" (UID: \"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" Jul 14 19:01:04.224751 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:04.224722 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" Jul 14 19:01:04.376805 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:04.376772 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct"] Jul 14 19:01:04.378085 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:01:04.378059 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f44bd1_b2c7_4e6d_83a0_9a9ab821fdf8.slice/crio-204adeb936d8ece878946447a3a6a8ba90b941818b0977fb5d8a279f395da7d0 WatchSource:0}: Error finding container 204adeb936d8ece878946447a3a6a8ba90b941818b0977fb5d8a279f395da7d0: Status 404 returned error can't find the container with id 204adeb936d8ece878946447a3a6a8ba90b941818b0977fb5d8a279f395da7d0 Jul 14 19:01:04.651850 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:04.651767 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" event={"ID":"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8","Type":"ContainerStarted","Data":"204adeb936d8ece878946447a3a6a8ba90b941818b0977fb5d8a279f395da7d0"} Jul 14 19:01:05.659060 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:05.659025 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" event={"ID":"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8","Type":"ContainerStarted","Data":"c6a00f8d237641cdc48bf5cb09ba77a213dae34725ee1160585c8dc0f5d5b618"} Jul 14 19:01:05.659490 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:05.659090 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" Jul 14 19:01:05.680394 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:05.680349 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" podStartSLOduration=2.100156965 podStartE2EDuration="2.680333642s" totalImagesPullingTime="580.176677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:01:05.678087683 +0000 UTC m=+2367.215535468" watchObservedRunningTime="2026-07-14 19:01:05.680333642 +0000 UTC m=+2367.217781427" Jul 14 19:01:05.956512 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:05.956478 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:06.055030 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:06.054989 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:08.341002 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:08.340965 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:08.842871 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:08.842834 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:09.159107 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:09.159015 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:09.462768 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:09.462737 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:10.583926 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:10.583886 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:10.681877 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:10.681843 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:11.670463 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:11.670435 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" Jul 14 19:01:11.847451 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:11.847413 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:11.965890 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:11.965852 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:12.089606 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.089565 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7"] Jul 14 19:01:12.093694 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.093668 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:01:12.096926 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.096909 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-601221-serving-cert\"" Jul 14 19:01:12.116715 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.116691 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7"] Jul 14 19:01:12.176457 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.176397 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:12.185560 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.185533 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02cd0dd-6a31-4aa2-867e-10ece5731b68-maas-api-tls\") pod \"maas-api-e2e-api-b-601221-84bc9b5988-zvbx7\" (UID: \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:01:12.185696 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.185629 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02cd0dd-6a31-4aa2-867e-10ece5731b68-openshift-service-ca\") pod \"maas-api-e2e-api-b-601221-84bc9b5988-zvbx7\" (UID: \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:01:12.185766 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.185744 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss25g\" (UniqueName: \"kubernetes.io/projected/b02cd0dd-6a31-4aa2-867e-10ece5731b68-kube-api-access-ss25g\") pod \"maas-api-e2e-api-b-601221-84bc9b5988-zvbx7\" (UID: \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:01:12.286657 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.286568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02cd0dd-6a31-4aa2-867e-10ece5731b68-maas-api-tls\") pod \"maas-api-e2e-api-b-601221-84bc9b5988-zvbx7\" (UID: \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:01:12.286657 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.286646 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02cd0dd-6a31-4aa2-867e-10ece5731b68-openshift-service-ca\") pod \"maas-api-e2e-api-b-601221-84bc9b5988-zvbx7\" (UID: \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:01:12.286886 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.286713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss25g\" (UniqueName: \"kubernetes.io/projected/b02cd0dd-6a31-4aa2-867e-10ece5731b68-kube-api-access-ss25g\") pod \"maas-api-e2e-api-b-601221-84bc9b5988-zvbx7\" (UID: \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:01:12.286886 ip-10-0-132-224 kubenswrapper[2573]: E0714 19:01:12.286716 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-601221-serving-cert: secret "maas-api-e2e-api-b-601221-serving-cert" not found Jul 14 19:01:12.286886 ip-10-0-132-224 kubenswrapper[2573]: E0714 19:01:12.286820 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b02cd0dd-6a31-4aa2-867e-10ece5731b68-maas-api-tls podName:b02cd0dd-6a31-4aa2-867e-10ece5731b68 nodeName:}" failed. No retries permitted until 2026-07-14 19:01:12.786801955 +0000 UTC m=+2374.324249718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b02cd0dd-6a31-4aa2-867e-10ece5731b68-maas-api-tls") pod "maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" (UID: "b02cd0dd-6a31-4aa2-867e-10ece5731b68") : secret "maas-api-e2e-api-b-601221-serving-cert" not found Jul 14 19:01:12.287408 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.287386 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02cd0dd-6a31-4aa2-867e-10ece5731b68-openshift-service-ca\") pod \"maas-api-e2e-api-b-601221-84bc9b5988-zvbx7\" (UID: \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:01:12.295555 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.295531 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss25g\" (UniqueName: \"kubernetes.io/projected/b02cd0dd-6a31-4aa2-867e-10ece5731b68-kube-api-access-ss25g\") pod \"maas-api-e2e-api-b-601221-84bc9b5988-zvbx7\" (UID: \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:01:12.608356 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.608254 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:12.745581 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.745539 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:12.791118 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.791090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02cd0dd-6a31-4aa2-867e-10ece5731b68-maas-api-tls\") pod \"maas-api-e2e-api-b-601221-84bc9b5988-zvbx7\" (UID: \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:01:12.793694 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.793669 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02cd0dd-6a31-4aa2-867e-10ece5731b68-maas-api-tls\") pod \"maas-api-e2e-api-b-601221-84bc9b5988-zvbx7\" (UID: \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:01:12.844780 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:12.844722 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:13.004212 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:13.004183 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:01:13.159737 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:13.159708 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7"] Jul 14 19:01:13.161482 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:01:13.161444 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02cd0dd_6a31_4aa2_867e_10ece5731b68.slice/crio-f29de9b7dbb17818697b3ffabefb93a992688445acb00ac95a9bc66488238e9a WatchSource:0}: Error finding container f29de9b7dbb17818697b3ffabefb93a992688445acb00ac95a9bc66488238e9a: Status 404 returned error can't find the container with id f29de9b7dbb17818697b3ffabefb93a992688445acb00ac95a9bc66488238e9a Jul 14 19:01:13.707895 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:13.707860 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" event={"ID":"b02cd0dd-6a31-4aa2-867e-10ece5731b68","Type":"ContainerStarted","Data":"45607637ea90aac16774ff8926c486af4b02f40423923cbd0449fd47b774e79b"} Jul 14 19:01:13.708064 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:13.707962 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" event={"ID":"b02cd0dd-6a31-4aa2-867e-10ece5731b68","Type":"ContainerStarted","Data":"f29de9b7dbb17818697b3ffabefb93a992688445acb00ac95a9bc66488238e9a"} Jul 14 19:01:13.708064 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:13.707991 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:01:13.735321 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:13.735257 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" podStartSLOduration=1.332588641 podStartE2EDuration="1.735242706s" totalImagesPullingTime="402.654065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:01:13.733527997 +0000 UTC m=+2375.270975782" watchObservedRunningTime="2026-07-14 19:01:13.735242706 +0000 UTC m=+2375.272690492" Jul 14 19:01:16.380006 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:16.379970 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:19.718826 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:19.718797 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:01:24.636969 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:24.636926 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82771c-j5rjm"] Jul 14 19:01:24.641280 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:24.641253 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82771c-j5rjm" Jul 14 19:01:24.644243 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:24.644218 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 19:01:24.648003 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:24.647978 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82771c-j5rjm"] Jul 14 19:01:24.712177 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:24.712149 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c864f7-e7c5-4206-8c3c-fd1efdc15056-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82771c-j5rjm\" (UID: \"c4c864f7-e7c5-4206-8c3c-fd1efdc15056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82771c-j5rjm" Jul 14 19:01:24.813626 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:24.813601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c864f7-e7c5-4206-8c3c-fd1efdc15056-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82771c-j5rjm\" (UID: \"c4c864f7-e7c5-4206-8c3c-fd1efdc15056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82771c-j5rjm" Jul 14 19:01:24.814187 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:24.814170 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c864f7-e7c5-4206-8c3c-fd1efdc15056-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-82771c-j5rjm\" (UID: \"c4c864f7-e7c5-4206-8c3c-fd1efdc15056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82771c-j5rjm" Jul 14 19:01:24.954531 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:24.954504 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82771c-j5rjm" Jul 14 19:01:25.127669 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:25.127617 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82771c-j5rjm"] Jul 14 19:01:25.130866 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:01:25.130833 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c864f7_e7c5_4206_8c3c_fd1efdc15056.slice/crio-ce49f98acad195cce25330edef243f4715e67f7dacedbe48a216b9a3c9651608 WatchSource:0}: Error finding container ce49f98acad195cce25330edef243f4715e67f7dacedbe48a216b9a3c9651608: Status 404 returned error can't find the container with id ce49f98acad195cce25330edef243f4715e67f7dacedbe48a216b9a3c9651608 Jul 14 19:01:25.270572 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:25.270486 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:25.370578 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:25.370544 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:25.779138 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:25.779102 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82771c-j5rjm" event={"ID":"c4c864f7-e7c5-4206-8c3c-fd1efdc15056","Type":"ContainerStarted","Data":"d6b40225fbb1fb9010a7404c1368ca830482352ee8e86f3b51f3f2d4f392aa0a"} Jul 14 19:01:25.779648 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:25.779202 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82771c-j5rjm" event={"ID":"c4c864f7-e7c5-4206-8c3c-fd1efdc15056","Type":"ContainerStarted","Data":"ce49f98acad195cce25330edef243f4715e67f7dacedbe48a216b9a3c9651608"} Jul 14 19:01:25.796492 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:25.796438 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82771c-j5rjm" podStartSLOduration=1.7964240280000001 podStartE2EDuration="1.796424028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:01:25.795513708 +0000 UTC m=+2387.332961494" watchObservedRunningTime="2026-07-14 19:01:25.796424028 +0000 UTC m=+2387.333871813" Jul 14 19:01:26.786988 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:26.786955 2573 generic.go:356] "Generic (PLEG): container finished" podID="c4c864f7-e7c5-4206-8c3c-fd1efdc15056" containerID="d6b40225fbb1fb9010a7404c1368ca830482352ee8e86f3b51f3f2d4f392aa0a" exitCode=0 Jul 14 19:01:26.787457 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:26.786992 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82771c-j5rjm" event={"ID":"c4c864f7-e7c5-4206-8c3c-fd1efdc15056","Type":"ContainerDied","Data":"d6b40225fbb1fb9010a7404c1368ca830482352ee8e86f3b51f3f2d4f392aa0a"} Jul 14 19:01:27.938522 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:27.938497 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82771c-j5rjm" Jul 14 19:01:28.048264 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:28.048170 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c864f7-e7c5-4206-8c3c-fd1efdc15056-maas-api-service-ca\") pod \"c4c864f7-e7c5-4206-8c3c-fd1efdc15056\" (UID: \"c4c864f7-e7c5-4206-8c3c-fd1efdc15056\") " Jul 14 19:01:28.048563 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:28.048539 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4c864f7-e7c5-4206-8c3c-fd1efdc15056-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4c864f7-e7c5-4206-8c3c-fd1efdc15056" (UID: "c4c864f7-e7c5-4206-8c3c-fd1efdc15056"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:01:28.149847 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:28.149805 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c864f7-e7c5-4206-8c3c-fd1efdc15056-maas-api-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:01:28.800207 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:28.800181 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82771c-j5rjm" Jul 14 19:01:28.800207 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:28.800192 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-82771c-j5rjm" event={"ID":"c4c864f7-e7c5-4206-8c3c-fd1efdc15056","Type":"ContainerDied","Data":"ce49f98acad195cce25330edef243f4715e67f7dacedbe48a216b9a3c9651608"} Jul 14 19:01:28.800446 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:28.800236 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce49f98acad195cce25330edef243f4715e67f7dacedbe48a216b9a3c9651608" Jul 14 19:01:33.596433 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:33.596398 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:33.690662 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:33.690631 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:35.346683 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:35.346647 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:35.453634 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:35.453590 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:35.639840 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:35.639758 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct"] Jul 14 19:01:35.640309 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:35.640033 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" podUID="94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8" containerName="maas-api" containerID="cri-o://c6a00f8d237641cdc48bf5cb09ba77a213dae34725ee1160585c8dc0f5d5b618" gracePeriod=30 Jul 14 19:01:35.672543 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:35.672508 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:35.844486 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:35.844447 2573 generic.go:356] "Generic (PLEG): container finished" podID="94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8" containerID="c6a00f8d237641cdc48bf5cb09ba77a213dae34725ee1160585c8dc0f5d5b618" exitCode=0 Jul 14 19:01:35.844633 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:35.844509 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" event={"ID":"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8","Type":"ContainerDied","Data":"c6a00f8d237641cdc48bf5cb09ba77a213dae34725ee1160585c8dc0f5d5b618"} Jul 14 19:01:35.907540 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:35.907474 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" Jul 14 19:01:36.026098 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:36.026068 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-597qd\" (UniqueName: \"kubernetes.io/projected/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-kube-api-access-597qd\") pod \"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8\" (UID: \"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8\") " Jul 14 19:01:36.026246 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:36.026106 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-openshift-service-ca\") pod \"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8\" (UID: \"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8\") " Jul 14 19:01:36.026246 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:36.026213 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-maas-api-tls\") pod \"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8\" (UID: \"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8\") " Jul 14 19:01:36.026485 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:36.026459 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8" (UID: "94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:01:36.026614 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:36.026586 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-openshift-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:01:36.028318 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:36.028270 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-kube-api-access-597qd" (OuterVolumeSpecName: "kube-api-access-597qd") pod "94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8" (UID: "94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8"). InnerVolumeSpecName "kube-api-access-597qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:01:36.028318 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:36.028262 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8" (UID: "94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:01:36.127142 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:36.127106 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-maas-api-tls\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:01:36.127142 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:36.127138 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-597qd\" (UniqueName: \"kubernetes.io/projected/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8-kube-api-access-597qd\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:01:36.852085 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:36.852057 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" Jul 14 19:01:36.852520 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:36.852093 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct" event={"ID":"94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8","Type":"ContainerDied","Data":"204adeb936d8ece878946447a3a6a8ba90b941818b0977fb5d8a279f395da7d0"} Jul 14 19:01:36.852520 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:36.852160 2573 scope.go:117] "RemoveContainer" containerID="c6a00f8d237641cdc48bf5cb09ba77a213dae34725ee1160585c8dc0f5d5b618" Jul 14 19:01:36.876618 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:36.876592 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct"] Jul 14 19:01:36.883067 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:36.883046 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-82771c-7d67c95767-59fct"] Jul 14 19:01:37.222459 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:37.222428 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8" path="/var/lib/kubelet/pods/94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8/volumes" Jul 14 19:01:39.090461 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:39.090427 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 19:01:39.098436 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:39.098414 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 19:01:39.099686 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:39.099669 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 19:01:39.106740 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:39.106726 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 19:01:39.743747 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:39.743710 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:46.118445 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:46.118412 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:46.219748 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:46.219711 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:50.513646 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:50.513581 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:50.765158 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:50.765060 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:50.886096 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:50.886061 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601221-7hqqx"] Jul 14 19:01:50.886569 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:50.886554 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8" containerName="maas-api" Jul 14 19:01:50.886622 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:50.886570 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8" containerName="maas-api" Jul 14 19:01:50.886622 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:50.886591 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4c864f7-e7c5-4206-8c3c-fd1efdc15056" containerName="revoke-keys" Jul 14 19:01:50.886622 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:50.886597 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4c864f7-e7c5-4206-8c3c-fd1efdc15056" containerName="revoke-keys" Jul 14 19:01:50.886715 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:50.886683 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="94f44bd1-b2c7-4e6d-83a0-9a9ab821fdf8" containerName="maas-api" Jul 14 19:01:50.886715 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:50.886690 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4c864f7-e7c5-4206-8c3c-fd1efdc15056" containerName="revoke-keys" Jul 14 19:01:50.889999 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:50.889982 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601221-7hqqx" Jul 14 19:01:50.893005 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:50.892985 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 19:01:50.897593 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:50.897568 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601221-7hqqx"] Jul 14 19:01:50.962885 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:50.962854 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa215b6-f4b1-4333-a8fa-7585b911cb22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-601221-7hqqx\" (UID: \"1aa215b6-f4b1-4333-a8fa-7585b911cb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601221-7hqqx" Jul 14 19:01:51.063997 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:51.063903 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa215b6-f4b1-4333-a8fa-7585b911cb22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-601221-7hqqx\" (UID: \"1aa215b6-f4b1-4333-a8fa-7585b911cb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601221-7hqqx" Jul 14 19:01:51.064633 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:51.064608 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa215b6-f4b1-4333-a8fa-7585b911cb22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-601221-7hqqx\" (UID: \"1aa215b6-f4b1-4333-a8fa-7585b911cb22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601221-7hqqx" Jul 14 19:01:51.203709 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:51.203677 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601221-7hqqx" Jul 14 19:01:51.350937 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:51.350909 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601221-7hqqx"] Jul 14 19:01:51.352521 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:01:51.352493 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa215b6_f4b1_4333_a8fa_7585b911cb22.slice/crio-7c80e6c42b90fded1886f8192a45997b871f709128af291de30bbaab23be4401 WatchSource:0}: Error finding container 7c80e6c42b90fded1886f8192a45997b871f709128af291de30bbaab23be4401: Status 404 returned error can't find the container with id 7c80e6c42b90fded1886f8192a45997b871f709128af291de30bbaab23be4401 Jul 14 19:01:51.442213 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:51.442182 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:51.542393 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:51.542361 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:01:51.945895 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:51.945864 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601221-7hqqx" event={"ID":"1aa215b6-f4b1-4333-a8fa-7585b911cb22","Type":"ContainerStarted","Data":"773c86d6c3d0fcf14af5b2da125a1ea76c1dba651e2c31e3eb323c187da2b6e3"} Jul 14 19:01:51.946078 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:51.945915 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601221-7hqqx" event={"ID":"1aa215b6-f4b1-4333-a8fa-7585b911cb22","Type":"ContainerStarted","Data":"7c80e6c42b90fded1886f8192a45997b871f709128af291de30bbaab23be4401"} Jul 14 19:01:51.965615 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:51.965571 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601221-7hqqx" podStartSLOduration=1.96555465 podStartE2EDuration="1.96555465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:01:51.962409702 +0000 UTC m=+2413.499857489" watchObservedRunningTime="2026-07-14 19:01:51.96555465 +0000 UTC m=+2413.503002435" Jul 14 19:01:52.953822 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:52.953789 2573 generic.go:356] "Generic (PLEG): container finished" podID="1aa215b6-f4b1-4333-a8fa-7585b911cb22" containerID="773c86d6c3d0fcf14af5b2da125a1ea76c1dba651e2c31e3eb323c187da2b6e3" exitCode=0 Jul 14 19:01:52.954210 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:52.953851 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601221-7hqqx" event={"ID":"1aa215b6-f4b1-4333-a8fa-7585b911cb22","Type":"ContainerDied","Data":"773c86d6c3d0fcf14af5b2da125a1ea76c1dba651e2c31e3eb323c187da2b6e3"} Jul 14 19:01:54.102523 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:54.102499 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601221-7hqqx" Jul 14 19:01:54.193858 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:54.193828 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa215b6-f4b1-4333-a8fa-7585b911cb22-maas-api-service-ca\") pod \"1aa215b6-f4b1-4333-a8fa-7585b911cb22\" (UID: \"1aa215b6-f4b1-4333-a8fa-7585b911cb22\") " Jul 14 19:01:54.194145 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:54.194121 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa215b6-f4b1-4333-a8fa-7585b911cb22-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1aa215b6-f4b1-4333-a8fa-7585b911cb22" (UID: "1aa215b6-f4b1-4333-a8fa-7585b911cb22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:01:54.294673 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:54.294603 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa215b6-f4b1-4333-a8fa-7585b911cb22-maas-api-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:01:54.967401 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:54.967373 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601221-7hqqx" Jul 14 19:01:54.967401 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:54.967381 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-601221-7hqqx" event={"ID":"1aa215b6-f4b1-4333-a8fa-7585b911cb22","Type":"ContainerDied","Data":"7c80e6c42b90fded1886f8192a45997b871f709128af291de30bbaab23be4401"} Jul 14 19:01:54.967606 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:01:54.967428 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c80e6c42b90fded1886f8192a45997b871f709128af291de30bbaab23be4401" Jul 14 19:02:01.366806 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:01.366732 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7"] Jul 14 19:02:01.367265 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:01.366964 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" podUID="b02cd0dd-6a31-4aa2-867e-10ece5731b68" containerName="maas-api" containerID="cri-o://45607637ea90aac16774ff8926c486af4b02f40423923cbd0449fd47b774e79b" gracePeriod=30 Jul 14 19:02:01.392227 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:01.392159 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:01.618963 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:01.618910 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:02:01.766661 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:01.766631 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss25g\" (UniqueName: \"kubernetes.io/projected/b02cd0dd-6a31-4aa2-867e-10ece5731b68-kube-api-access-ss25g\") pod \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\" (UID: \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\") " Jul 14 19:02:01.766828 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:01.766760 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02cd0dd-6a31-4aa2-867e-10ece5731b68-openshift-service-ca\") pod \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\" (UID: \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\") " Jul 14 19:02:01.766828 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:01.766798 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02cd0dd-6a31-4aa2-867e-10ece5731b68-maas-api-tls\") pod \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\" (UID: \"b02cd0dd-6a31-4aa2-867e-10ece5731b68\") " Jul 14 19:02:01.767108 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:01.767075 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02cd0dd-6a31-4aa2-867e-10ece5731b68-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b02cd0dd-6a31-4aa2-867e-10ece5731b68" (UID: "b02cd0dd-6a31-4aa2-867e-10ece5731b68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:02:01.767245 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:01.767156 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02cd0dd-6a31-4aa2-867e-10ece5731b68-openshift-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:02:01.768776 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:01.768746 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02cd0dd-6a31-4aa2-867e-10ece5731b68-kube-api-access-ss25g" (OuterVolumeSpecName: "kube-api-access-ss25g") pod "b02cd0dd-6a31-4aa2-867e-10ece5731b68" (UID: "b02cd0dd-6a31-4aa2-867e-10ece5731b68"). InnerVolumeSpecName "kube-api-access-ss25g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:02:01.768849 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:01.768813 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02cd0dd-6a31-4aa2-867e-10ece5731b68-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b02cd0dd-6a31-4aa2-867e-10ece5731b68" (UID: "b02cd0dd-6a31-4aa2-867e-10ece5731b68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:02:01.868276 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:01.868247 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02cd0dd-6a31-4aa2-867e-10ece5731b68-maas-api-tls\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:02:01.868276 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:01.868272 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss25g\" (UniqueName: \"kubernetes.io/projected/b02cd0dd-6a31-4aa2-867e-10ece5731b68-kube-api-access-ss25g\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:02:02.010346 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:02.010313 2573 generic.go:356] "Generic (PLEG): container finished" podID="b02cd0dd-6a31-4aa2-867e-10ece5731b68" containerID="45607637ea90aac16774ff8926c486af4b02f40423923cbd0449fd47b774e79b" exitCode=0 Jul 14 19:02:02.010512 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:02.010348 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" Jul 14 19:02:02.010512 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:02.010379 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" event={"ID":"b02cd0dd-6a31-4aa2-867e-10ece5731b68","Type":"ContainerDied","Data":"45607637ea90aac16774ff8926c486af4b02f40423923cbd0449fd47b774e79b"} Jul 14 19:02:02.010512 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:02.010442 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7" event={"ID":"b02cd0dd-6a31-4aa2-867e-10ece5731b68","Type":"ContainerDied","Data":"f29de9b7dbb17818697b3ffabefb93a992688445acb00ac95a9bc66488238e9a"} Jul 14 19:02:02.010512 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:02.010471 2573 scope.go:117] "RemoveContainer" containerID="45607637ea90aac16774ff8926c486af4b02f40423923cbd0449fd47b774e79b" Jul 14 19:02:02.021419 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:02.021401 2573 scope.go:117] "RemoveContainer" containerID="45607637ea90aac16774ff8926c486af4b02f40423923cbd0449fd47b774e79b" Jul 14 19:02:02.021690 ip-10-0-132-224 kubenswrapper[2573]: E0714 19:02:02.021671 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45607637ea90aac16774ff8926c486af4b02f40423923cbd0449fd47b774e79b\": container with ID starting with 45607637ea90aac16774ff8926c486af4b02f40423923cbd0449fd47b774e79b not found: ID does not exist" containerID="45607637ea90aac16774ff8926c486af4b02f40423923cbd0449fd47b774e79b" Jul 14 19:02:02.021763 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:02.021699 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45607637ea90aac16774ff8926c486af4b02f40423923cbd0449fd47b774e79b"} err="failed to get container status \"45607637ea90aac16774ff8926c486af4b02f40423923cbd0449fd47b774e79b\": rpc error: code = NotFound desc = could not find container \"45607637ea90aac16774ff8926c486af4b02f40423923cbd0449fd47b774e79b\": container with ID starting with 45607637ea90aac16774ff8926c486af4b02f40423923cbd0449fd47b774e79b not found: ID does not exist" Jul 14 19:02:02.035353 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:02.035330 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7"] Jul 14 19:02:02.043617 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:02.043597 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-601221-84bc9b5988-zvbx7"] Jul 14 19:02:03.222476 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:03.222448 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b02cd0dd-6a31-4aa2-867e-10ece5731b68" path="/var/lib/kubelet/pods/b02cd0dd-6a31-4aa2-867e-10ece5731b68/volumes" Jul 14 19:02:05.984556 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:05.984519 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:12.389115 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:12.389080 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:12.487118 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:12.487061 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:16.727760 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:16.727709 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:16.984071 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:16.983979 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:17.369747 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:17.369657 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:17.694559 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:17.694521 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:17.906458 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:17.906414 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:21.121819 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.121787 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:21.209183 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.209149 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4"] Jul 14 19:02:21.209686 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.209671 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b02cd0dd-6a31-4aa2-867e-10ece5731b68" containerName="maas-api" Jul 14 19:02:21.209738 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.209688 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b02cd0dd-6a31-4aa2-867e-10ece5731b68" containerName="maas-api" Jul 14 19:02:21.209738 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.209706 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aa215b6-f4b1-4333-a8fa-7585b911cb22" containerName="revoke-keys" Jul 14 19:02:21.209738 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.209713 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aa215b6-f4b1-4333-a8fa-7585b911cb22" containerName="revoke-keys" Jul 14 19:02:21.209832 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.209817 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b02cd0dd-6a31-4aa2-867e-10ece5731b68" containerName="maas-api" Jul 14 19:02:21.209865 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.209829 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aa215b6-f4b1-4333-a8fa-7585b911cb22" containerName="revoke-keys" Jul 14 19:02:21.213136 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.213118 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:02:21.216319 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.216298 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-52ef23-serving-cert\"" Jul 14 19:02:21.227190 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.227150 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4"] Jul 14 19:02:21.228163 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.228136 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:21.255417 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.255391 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4\" (UID: \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:02:21.255542 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.255443 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkt87\" (UniqueName: \"kubernetes.io/projected/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-kube-api-access-qkt87\") pod \"maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4\" (UID: \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:02:21.255587 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.255545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4\" (UID: \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:02:21.333825 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.333791 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:21.356231 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.356203 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4\" (UID: \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:02:21.356400 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.356254 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkt87\" (UniqueName: \"kubernetes.io/projected/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-kube-api-access-qkt87\") pod \"maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4\" (UID: \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:02:21.356400 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.356335 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4\" (UID: \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:02:21.356546 ip-10-0-132-224 kubenswrapper[2573]: E0714 19:02:21.356530 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-serving-cert: secret "maas-api-e2e-shared-a-52ef23-serving-cert" not found Jul 14 19:02:21.356621 ip-10-0-132-224 kubenswrapper[2573]: E0714 19:02:21.356606 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-maas-api-tls podName:6c215e7c-39ec-4a65-aeab-a67c2752c2f9 nodeName:}" failed. No retries permitted until 2026-07-14 19:02:21.856584413 +0000 UTC m=+2443.394032195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-maas-api-tls") pod "maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" (UID: "6c215e7c-39ec-4a65-aeab-a67c2752c2f9") : secret "maas-api-e2e-shared-a-52ef23-serving-cert" not found Jul 14 19:02:21.357002 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.356983 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4\" (UID: \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:02:21.379069 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.379008 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkt87\" (UniqueName: \"kubernetes.io/projected/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-kube-api-access-qkt87\") pod \"maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4\" (UID: \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:02:21.429184 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.429153 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:21.860253 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.860219 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4\" (UID: \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:02:21.862682 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.862645 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4\" (UID: \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:02:21.948997 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:21.948961 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:22.010086 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:22.010043 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:22.124216 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:22.124122 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:02:22.275377 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:22.275342 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4"] Jul 14 19:02:22.275850 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:02:22.275827 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c215e7c_39ec_4a65_aeab_a67c2752c2f9.slice/crio-ae9c6b31163abf6db759c49346fe07237b291fb7b3b95bd70060d82684249f78 WatchSource:0}: Error finding container ae9c6b31163abf6db759c49346fe07237b291fb7b3b95bd70060d82684249f78: Status 404 returned error can't find the container with id ae9c6b31163abf6db759c49346fe07237b291fb7b3b95bd70060d82684249f78 Jul 14 19:02:22.277160 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:22.277142 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:02:23.152358 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:23.152255 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" event={"ID":"6c215e7c-39ec-4a65-aeab-a67c2752c2f9","Type":"ContainerStarted","Data":"2c1d46087c5379ff3b17991f959a6d0b8d4594766c88b1f01286acf708abb3fc"} Jul 14 19:02:23.152710 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:23.152468 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:02:23.152710 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:23.152515 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" event={"ID":"6c215e7c-39ec-4a65-aeab-a67c2752c2f9","Type":"ContainerStarted","Data":"ae9c6b31163abf6db759c49346fe07237b291fb7b3b95bd70060d82684249f78"} Jul 14 19:02:23.172106 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:23.172066 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" podStartSLOduration=1.5286272140000001 podStartE2EDuration="2.172053073s" totalImagesPullingTime="643.425859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:02:23.170367401 +0000 UTC m=+2444.707815196" watchObservedRunningTime="2026-07-14 19:02:23.172053073 +0000 UTC m=+2444.709500855" Jul 14 19:02:25.699679 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:25.699647 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:26.036597 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:26.036505 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:26.373455 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:26.373364 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:26.548312 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:26.548250 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:29.164565 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.164536 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:02:29.211623 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.211579 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:29.294436 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.294405 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt"] Jul 14 19:02:29.297675 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.297643 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:29.298652 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.298636 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" Jul 14 19:02:29.301458 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.301432 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-10fa01-serving-cert\"" Jul 14 19:02:29.312954 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.312931 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt"] Jul 14 19:02:29.320632 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.320607 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da363f37-749c-409a-adef-710c875563ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt\" (UID: \"da363f37-749c-409a-adef-710c875563ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" Jul 14 19:02:29.320744 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.320644 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da363f37-749c-409a-adef-710c875563ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt\" (UID: \"da363f37-749c-409a-adef-710c875563ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" Jul 14 19:02:29.320744 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.320721 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r7lm\" (UniqueName: \"kubernetes.io/projected/da363f37-749c-409a-adef-710c875563ff-kube-api-access-4r7lm\") pod \"maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt\" (UID: \"da363f37-749c-409a-adef-710c875563ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" Jul 14 19:02:29.421529 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.421440 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da363f37-749c-409a-adef-710c875563ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt\" (UID: \"da363f37-749c-409a-adef-710c875563ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" Jul 14 19:02:29.421529 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.421485 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da363f37-749c-409a-adef-710c875563ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt\" (UID: \"da363f37-749c-409a-adef-710c875563ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" Jul 14 19:02:29.421529 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.421522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r7lm\" (UniqueName: \"kubernetes.io/projected/da363f37-749c-409a-adef-710c875563ff-kube-api-access-4r7lm\") pod \"maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt\" (UID: \"da363f37-749c-409a-adef-710c875563ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" Jul 14 19:02:29.422164 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.422140 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da363f37-749c-409a-adef-710c875563ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt\" (UID: \"da363f37-749c-409a-adef-710c875563ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" Jul 14 19:02:29.423855 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.423822 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da363f37-749c-409a-adef-710c875563ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt\" (UID: \"da363f37-749c-409a-adef-710c875563ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" Jul 14 19:02:29.428017 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.427991 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:29.433551 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.433533 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r7lm\" (UniqueName: \"kubernetes.io/projected/da363f37-749c-409a-adef-710c875563ff-kube-api-access-4r7lm\") pod \"maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt\" (UID: \"da363f37-749c-409a-adef-710c875563ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" Jul 14 19:02:29.578247 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.578215 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:29.610407 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.610380 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" Jul 14 19:02:29.759220 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.759196 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt"] Jul 14 19:02:29.761054 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:02:29.761021 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda363f37_749c_409a_adef_710c875563ff.slice/crio-eb8c1831c1b17a63cbe03451f6ef022215ea426898422ec02fb93674d1421a4d WatchSource:0}: Error finding container eb8c1831c1b17a63cbe03451f6ef022215ea426898422ec02fb93674d1421a4d: Status 404 returned error can't find the container with id eb8c1831c1b17a63cbe03451f6ef022215ea426898422ec02fb93674d1421a4d Jul 14 19:02:29.990853 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:29.990764 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:30.090015 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:30.089975 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:30.196174 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:30.196142 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" event={"ID":"da363f37-749c-409a-adef-710c875563ff","Type":"ContainerStarted","Data":"13568f88ffc6a864d275b2a4f46ed4cf14a78047ccacfb78eb47a4835d0c88de"} Jul 14 19:02:30.196577 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:30.196193 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" event={"ID":"da363f37-749c-409a-adef-710c875563ff","Type":"ContainerStarted","Data":"eb8c1831c1b17a63cbe03451f6ef022215ea426898422ec02fb93674d1421a4d"} Jul 14 19:02:30.196577 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:30.196270 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" Jul 14 19:02:30.216128 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:30.216033 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" podStartSLOduration=0.876181043 podStartE2EDuration="1.216012562s" totalImagesPullingTime="339.831519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:02:30.214898849 +0000 UTC m=+2451.752346633" watchObservedRunningTime="2026-07-14 19:02:30.216012562 +0000 UTC m=+2451.753460347" Jul 14 19:02:33.770563 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:33.770509 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:36.208451 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:36.208419 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" Jul 14 19:02:39.212888 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:39.212852 2573 scope.go:117] "RemoveContainer" containerID="c120c0a390ae6b41be19b0ab5412b176e529874b6eb42e7b957f883f1fb601cb" Jul 14 19:02:42.775516 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:42.775481 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:44.023034 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:44.022999 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:45.915475 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:45.915439 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:50.878124 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:50.878069 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:02:50.981915 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:02:50.981879 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:03:10.077750 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:03:10.077713 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:03:26.292427 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:03:26.292386 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:03:43.155253 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:03:43.155167 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:03:44.282178 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:03:44.282138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:05:12.727754 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:05:12.727708 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:05:13.254407 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:05:13.254372 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:05:14.110840 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:05:14.110805 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:05:19.584658 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:05:19.584623 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:05:35.265820 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:05:35.265782 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:06:07.230574 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:06:07.230538 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:06:11.429984 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:06:11.429938 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:06:11.533124 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:06:11.533078 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:06:39.131753 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:06:39.131667 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 19:06:39.139347 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:06:39.139327 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 19:06:39.141691 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:06:39.141670 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 19:06:39.148548 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:06:39.148532 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8pxlk_c42659ad-83fd-453f-afe1-39f0b9396257/ovn-acl-logging/0.log" Jul 14 19:06:39.264627 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:06:39.264593 2573 scope.go:117] "RemoveContainer" containerID="ccaadf164738bc7f1f7786ad6c8b35a5b72bc73b7b729be242d64d10edb87227" Jul 14 19:06:39.273613 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:06:39.273588 2573 scope.go:117] "RemoveContainer" containerID="3d73ced0d5690541e9bac1dc37cdb1dc04dbe64b35c398cecf277f86d89340f2" Jul 14 19:06:48.322941 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:06:48.322909 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:06:49.110179 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:06:49.110138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:06:56.984650 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:06:56.984612 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:07:25.310455 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:07:25.310415 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:07:43.824345 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:07:43.824300 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:07:43.931947 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:07:43.931910 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:07:51.816772 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:07:51.816731 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:07:52.590594 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:07:52.590550 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:08:08.138895 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:08.138852 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:08:09.020839 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:09.020796 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:08:10.055441 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:10.055401 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7944-139954956136112"] Jul 14 19:08:10.059534 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:10.059510 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954956136112" Jul 14 19:08:10.065143 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:10.065120 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-mqq4m\"" Jul 14 19:08:10.068787 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:10.068765 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7944-139954956136112"] Jul 14 19:08:10.076003 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:10.075982 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvhhf\" (UniqueName: \"kubernetes.io/projected/80e6a4fb-66b2-4d40-8c40-1a5381753a7e-kube-api-access-dvhhf\") pod \"test-curl-7944-139954956136112\" (UID: \"80e6a4fb-66b2-4d40-8c40-1a5381753a7e\") " pod="openshift-ingress/test-curl-7944-139954956136112" Jul 14 19:08:10.176631 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:10.176600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvhhf\" (UniqueName: \"kubernetes.io/projected/80e6a4fb-66b2-4d40-8c40-1a5381753a7e-kube-api-access-dvhhf\") pod \"test-curl-7944-139954956136112\" (UID: \"80e6a4fb-66b2-4d40-8c40-1a5381753a7e\") " pod="openshift-ingress/test-curl-7944-139954956136112" Jul 14 19:08:10.189250 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:10.189224 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvhhf\" (UniqueName: \"kubernetes.io/projected/80e6a4fb-66b2-4d40-8c40-1a5381753a7e-kube-api-access-dvhhf\") pod \"test-curl-7944-139954956136112\" (UID: \"80e6a4fb-66b2-4d40-8c40-1a5381753a7e\") " pod="openshift-ingress/test-curl-7944-139954956136112" Jul 14 19:08:10.369812 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:10.369737 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954956136112" Jul 14 19:08:10.713379 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:10.713351 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7944-139954956136112"] Jul 14 19:08:10.714662 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:08:10.714634 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80e6a4fb_66b2_4d40_8c40_1a5381753a7e.slice/crio-f30567e915be4f22c294bd5d59589fd2a09706f17a1330b02d385871581c77c6 WatchSource:0}: Error finding container f30567e915be4f22c294bd5d59589fd2a09706f17a1330b02d385871581c77c6: Status 404 returned error can't find the container with id f30567e915be4f22c294bd5d59589fd2a09706f17a1330b02d385871581c77c6 Jul 14 19:08:10.716127 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:10.716109 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:08:11.261034 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:11.260988 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7944-139954956136112" event={"ID":"80e6a4fb-66b2-4d40-8c40-1a5381753a7e","Type":"ContainerStarted","Data":"f30567e915be4f22c294bd5d59589fd2a09706f17a1330b02d385871581c77c6"} Jul 14 19:08:12.268527 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:12.268495 2573 generic.go:356] "Generic (PLEG): container finished" podID="80e6a4fb-66b2-4d40-8c40-1a5381753a7e" containerID="abfc88852fe1db3e40c06af7d5e59167436255a5d35a510a60fe44b3b47aac0d" exitCode=0 Jul 14 19:08:12.268887 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:12.268563 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7944-139954956136112" event={"ID":"80e6a4fb-66b2-4d40-8c40-1a5381753a7e","Type":"ContainerDied","Data":"abfc88852fe1db3e40c06af7d5e59167436255a5d35a510a60fe44b3b47aac0d"} Jul 14 19:08:13.425825 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.425802 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954956136112" Jul 14 19:08:13.450124 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.450096 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7944-139954956136112_80e6a4fb-66b2-4d40-8c40-1a5381753a7e/test-curl-7944-139954956136112/0.log" Jul 14 19:08:13.482206 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.482176 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7944-139954956136112"] Jul 14 19:08:13.490135 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.490115 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7944-139954956136112"] Jul 14 19:08:13.507763 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.507744 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvhhf\" (UniqueName: \"kubernetes.io/projected/80e6a4fb-66b2-4d40-8c40-1a5381753a7e-kube-api-access-dvhhf\") pod \"80e6a4fb-66b2-4d40-8c40-1a5381753a7e\" (UID: \"80e6a4fb-66b2-4d40-8c40-1a5381753a7e\") " Jul 14 19:08:13.509702 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.509676 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e6a4fb-66b2-4d40-8c40-1a5381753a7e-kube-api-access-dvhhf" (OuterVolumeSpecName: "kube-api-access-dvhhf") pod "80e6a4fb-66b2-4d40-8c40-1a5381753a7e" (UID: "80e6a4fb-66b2-4d40-8c40-1a5381753a7e"). InnerVolumeSpecName "kube-api-access-dvhhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:08:13.609136 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.609072 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvhhf\" (UniqueName: \"kubernetes.io/projected/80e6a4fb-66b2-4d40-8c40-1a5381753a7e-kube-api-access-dvhhf\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:08:13.628745 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.628723 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7944-139954953671856"] Jul 14 19:08:13.629211 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.629199 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e6a4fb-66b2-4d40-8c40-1a5381753a7e" containerName="test-curl-7944-139954956136112" Jul 14 19:08:13.629254 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.629213 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e6a4fb-66b2-4d40-8c40-1a5381753a7e" containerName="test-curl-7944-139954956136112" Jul 14 19:08:13.629340 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.629329 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e6a4fb-66b2-4d40-8c40-1a5381753a7e" containerName="test-curl-7944-139954956136112" Jul 14 19:08:13.632583 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.632567 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953671856" Jul 14 19:08:13.641495 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.641472 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7944-139954953671856"] Jul 14 19:08:13.710370 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.710339 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjhzv\" (UniqueName: \"kubernetes.io/projected/61414391-c35d-4b40-a553-4450e95fce10-kube-api-access-hjhzv\") pod \"test-curl-7944-139954953671856\" (UID: \"61414391-c35d-4b40-a553-4450e95fce10\") " pod="openshift-ingress/test-curl-7944-139954953671856" Jul 14 19:08:13.811236 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.811202 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjhzv\" (UniqueName: \"kubernetes.io/projected/61414391-c35d-4b40-a553-4450e95fce10-kube-api-access-hjhzv\") pod \"test-curl-7944-139954953671856\" (UID: \"61414391-c35d-4b40-a553-4450e95fce10\") " pod="openshift-ingress/test-curl-7944-139954953671856" Jul 14 19:08:13.820552 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.820532 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjhzv\" (UniqueName: \"kubernetes.io/projected/61414391-c35d-4b40-a553-4450e95fce10-kube-api-access-hjhzv\") pod \"test-curl-7944-139954953671856\" (UID: \"61414391-c35d-4b40-a553-4450e95fce10\") " pod="openshift-ingress/test-curl-7944-139954953671856" Jul 14 19:08:13.942404 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:13.942370 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953671856" Jul 14 19:08:14.085195 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:14.085168 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7944-139954953671856"] Jul 14 19:08:14.086528 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:08:14.086500 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61414391_c35d_4b40_a553_4450e95fce10.slice/crio-fc839e5c86fb17cbc74b48db3c28a089434ec532e3456daa4d41d948fbbc2c3a WatchSource:0}: Error finding container fc839e5c86fb17cbc74b48db3c28a089434ec532e3456daa4d41d948fbbc2c3a: Status 404 returned error can't find the container with id fc839e5c86fb17cbc74b48db3c28a089434ec532e3456daa4d41d948fbbc2c3a Jul 14 19:08:14.281181 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:14.281087 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7944-139954953671856" event={"ID":"61414391-c35d-4b40-a553-4450e95fce10","Type":"ContainerStarted","Data":"fc839e5c86fb17cbc74b48db3c28a089434ec532e3456daa4d41d948fbbc2c3a"} Jul 14 19:08:14.282273 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:14.282256 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954956136112" Jul 14 19:08:14.282399 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:14.282315 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f30567e915be4f22c294bd5d59589fd2a09706f17a1330b02d385871581c77c6" Jul 14 19:08:14.293359 ip-10-0-132-224 kubenswrapper[2573]: E0714 19:08:14.293334 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7944-139954956136112\" is forbidden: User \"system:node:ip-10-0-132-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-132-224.ec2.internal' and this object" podUID="80e6a4fb-66b2-4d40-8c40-1a5381753a7e" pod="openshift-ingress/test-curl-7944-139954956136112" Jul 14 19:08:15.223167 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:15.223131 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80e6a4fb-66b2-4d40-8c40-1a5381753a7e" path="/var/lib/kubelet/pods/80e6a4fb-66b2-4d40-8c40-1a5381753a7e/volumes" Jul 14 19:08:15.296308 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:15.290342 2573 generic.go:356] "Generic (PLEG): container finished" podID="61414391-c35d-4b40-a553-4450e95fce10" containerID="5a3f92af65d26996af64ad849cae1c526064fc52741345e41f0ecef074b5749d" exitCode=0 Jul 14 19:08:15.296308 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:15.290411 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7944-139954953671856" event={"ID":"61414391-c35d-4b40-a553-4450e95fce10","Type":"ContainerDied","Data":"5a3f92af65d26996af64ad849cae1c526064fc52741345e41f0ecef074b5749d"} Jul 14 19:08:16.460457 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.460433 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953671856" Jul 14 19:08:16.482836 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.482810 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7944-139954953671856_61414391-c35d-4b40-a553-4450e95fce10/test-curl-7944-139954953671856/0.log" Jul 14 19:08:16.510236 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.510209 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7944-139954953671856"] Jul 14 19:08:16.513774 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.513753 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7944-139954953671856"] Jul 14 19:08:16.536470 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.536449 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjhzv\" (UniqueName: \"kubernetes.io/projected/61414391-c35d-4b40-a553-4450e95fce10-kube-api-access-hjhzv\") pod \"61414391-c35d-4b40-a553-4450e95fce10\" (UID: \"61414391-c35d-4b40-a553-4450e95fce10\") " Jul 14 19:08:16.538380 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.538355 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61414391-c35d-4b40-a553-4450e95fce10-kube-api-access-hjhzv" (OuterVolumeSpecName: "kube-api-access-hjhzv") pod "61414391-c35d-4b40-a553-4450e95fce10" (UID: "61414391-c35d-4b40-a553-4450e95fce10"). InnerVolumeSpecName "kube-api-access-hjhzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:08:16.637571 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.637476 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjhzv\" (UniqueName: \"kubernetes.io/projected/61414391-c35d-4b40-a553-4450e95fce10-kube-api-access-hjhzv\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:08:16.687666 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.687630 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7944-139954953695280"] Jul 14 19:08:16.688170 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.688156 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61414391-c35d-4b40-a553-4450e95fce10" containerName="test-curl-7944-139954953671856" Jul 14 19:08:16.688228 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.688171 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="61414391-c35d-4b40-a553-4450e95fce10" containerName="test-curl-7944-139954953671856" Jul 14 19:08:16.688266 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.688244 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="61414391-c35d-4b40-a553-4450e95fce10" containerName="test-curl-7944-139954953671856" Jul 14 19:08:16.691539 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.691520 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953695280" Jul 14 19:08:16.698647 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.698621 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7944-139954953695280"] Jul 14 19:08:16.738914 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.738886 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp5pk\" (UniqueName: \"kubernetes.io/projected/bbf15e9f-541d-4ac7-a4fe-bf2496e45773-kube-api-access-pp5pk\") pod \"test-curl-7944-139954953695280\" (UID: \"bbf15e9f-541d-4ac7-a4fe-bf2496e45773\") " pod="openshift-ingress/test-curl-7944-139954953695280" Jul 14 19:08:16.840337 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.840272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp5pk\" (UniqueName: \"kubernetes.io/projected/bbf15e9f-541d-4ac7-a4fe-bf2496e45773-kube-api-access-pp5pk\") pod \"test-curl-7944-139954953695280\" (UID: \"bbf15e9f-541d-4ac7-a4fe-bf2496e45773\") " pod="openshift-ingress/test-curl-7944-139954953695280" Jul 14 19:08:16.850233 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:16.850207 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp5pk\" (UniqueName: \"kubernetes.io/projected/bbf15e9f-541d-4ac7-a4fe-bf2496e45773-kube-api-access-pp5pk\") pod \"test-curl-7944-139954953695280\" (UID: \"bbf15e9f-541d-4ac7-a4fe-bf2496e45773\") " pod="openshift-ingress/test-curl-7944-139954953695280" Jul 14 19:08:17.002312 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:17.002262 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953695280" Jul 14 19:08:17.145078 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:17.145056 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7944-139954953695280"] Jul 14 19:08:17.146490 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:08:17.146466 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbf15e9f_541d_4ac7_a4fe_bf2496e45773.slice/crio-574a4986072a8e6fe97dd1bafcb88be0af808f064d5e3e8141a8fec176e5b57f WatchSource:0}: Error finding container 574a4986072a8e6fe97dd1bafcb88be0af808f064d5e3e8141a8fec176e5b57f: Status 404 returned error can't find the container with id 574a4986072a8e6fe97dd1bafcb88be0af808f064d5e3e8141a8fec176e5b57f Jul 14 19:08:17.223103 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:17.223071 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61414391-c35d-4b40-a553-4450e95fce10" path="/var/lib/kubelet/pods/61414391-c35d-4b40-a553-4450e95fce10/volumes" Jul 14 19:08:17.303748 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:17.303676 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7944-139954953695280" event={"ID":"bbf15e9f-541d-4ac7-a4fe-bf2496e45773","Type":"ContainerStarted","Data":"574a4986072a8e6fe97dd1bafcb88be0af808f064d5e3e8141a8fec176e5b57f"} Jul 14 19:08:17.304976 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:17.304957 2573 scope.go:117] "RemoveContainer" containerID="5a3f92af65d26996af64ad849cae1c526064fc52741345e41f0ecef074b5749d" Jul 14 19:08:17.305083 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:17.304982 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953671856" Jul 14 19:08:18.313835 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:18.313806 2573 generic.go:356] "Generic (PLEG): container finished" podID="bbf15e9f-541d-4ac7-a4fe-bf2496e45773" containerID="430f23f073020b318c53d0bc116386ac081600c48e49850a839110da6aeaa24d" exitCode=0 Jul 14 19:08:18.314222 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:18.313892 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7944-139954953695280" event={"ID":"bbf15e9f-541d-4ac7-a4fe-bf2496e45773","Type":"ContainerDied","Data":"430f23f073020b318c53d0bc116386ac081600c48e49850a839110da6aeaa24d"} Jul 14 19:08:19.483221 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.483196 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953695280" Jul 14 19:08:19.507911 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.507879 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7944-139954953695280_bbf15e9f-541d-4ac7-a4fe-bf2496e45773/test-curl-7944-139954953695280/0.log" Jul 14 19:08:19.536534 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.536502 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7944-139954953695280"] Jul 14 19:08:19.540827 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.540805 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7944-139954953695280"] Jul 14 19:08:19.565749 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.565727 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp5pk\" (UniqueName: \"kubernetes.io/projected/bbf15e9f-541d-4ac7-a4fe-bf2496e45773-kube-api-access-pp5pk\") pod \"bbf15e9f-541d-4ac7-a4fe-bf2496e45773\" (UID: \"bbf15e9f-541d-4ac7-a4fe-bf2496e45773\") " Jul 14 19:08:19.567725 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.567703 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf15e9f-541d-4ac7-a4fe-bf2496e45773-kube-api-access-pp5pk" (OuterVolumeSpecName: "kube-api-access-pp5pk") pod "bbf15e9f-541d-4ac7-a4fe-bf2496e45773" (UID: "bbf15e9f-541d-4ac7-a4fe-bf2496e45773"). InnerVolumeSpecName "kube-api-access-pp5pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:08:19.667119 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.667095 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp5pk\" (UniqueName: \"kubernetes.io/projected/bbf15e9f-541d-4ac7-a4fe-bf2496e45773-kube-api-access-pp5pk\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:08:19.679248 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.679225 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7944-139954953695280"] Jul 14 19:08:19.679737 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.679723 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbf15e9f-541d-4ac7-a4fe-bf2496e45773" containerName="test-curl-7944-139954953695280" Jul 14 19:08:19.679782 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.679739 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbf15e9f-541d-4ac7-a4fe-bf2496e45773" containerName="test-curl-7944-139954953695280" Jul 14 19:08:19.679822 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.679812 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbf15e9f-541d-4ac7-a4fe-bf2496e45773" containerName="test-curl-7944-139954953695280" Jul 14 19:08:19.682982 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.682965 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953695280" Jul 14 19:08:19.693410 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.693385 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7944-139954953695280"] Jul 14 19:08:19.768396 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.768366 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxr9l\" (UniqueName: \"kubernetes.io/projected/ae85070a-1d2a-48cd-a14a-e3fbf927c61f-kube-api-access-sxr9l\") pod \"test-curl-7944-139954953695280\" (UID: \"ae85070a-1d2a-48cd-a14a-e3fbf927c61f\") " pod="openshift-ingress/test-curl-7944-139954953695280" Jul 14 19:08:19.869752 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.869666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxr9l\" (UniqueName: \"kubernetes.io/projected/ae85070a-1d2a-48cd-a14a-e3fbf927c61f-kube-api-access-sxr9l\") pod \"test-curl-7944-139954953695280\" (UID: \"ae85070a-1d2a-48cd-a14a-e3fbf927c61f\") " pod="openshift-ingress/test-curl-7944-139954953695280" Jul 14 19:08:19.879623 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.879596 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxr9l\" (UniqueName: \"kubernetes.io/projected/ae85070a-1d2a-48cd-a14a-e3fbf927c61f-kube-api-access-sxr9l\") pod \"test-curl-7944-139954953695280\" (UID: \"ae85070a-1d2a-48cd-a14a-e3fbf927c61f\") " pod="openshift-ingress/test-curl-7944-139954953695280" Jul 14 19:08:19.998031 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:19.998007 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953695280" Jul 14 19:08:20.144232 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:20.144199 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7944-139954953695280"] Jul 14 19:08:20.144981 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:08:20.144954 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae85070a_1d2a_48cd_a14a_e3fbf927c61f.slice/crio-81d07af4e3fcc2150a778fb2e2f1b29f9e9bc880a9df1e19523c96cbee003745 WatchSource:0}: Error finding container 81d07af4e3fcc2150a778fb2e2f1b29f9e9bc880a9df1e19523c96cbee003745: Status 404 returned error can't find the container with id 81d07af4e3fcc2150a778fb2e2f1b29f9e9bc880a9df1e19523c96cbee003745 Jul 14 19:08:20.328560 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:20.328516 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7944-139954953695280" event={"ID":"ae85070a-1d2a-48cd-a14a-e3fbf927c61f","Type":"ContainerStarted","Data":"81d07af4e3fcc2150a778fb2e2f1b29f9e9bc880a9df1e19523c96cbee003745"} Jul 14 19:08:20.329715 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:20.329694 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="574a4986072a8e6fe97dd1bafcb88be0af808f064d5e3e8141a8fec176e5b57f" Jul 14 19:08:20.329848 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:20.329746 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953695280" Jul 14 19:08:20.341830 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:20.341802 2573 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7944-139954953695280" oldPodUID="bbf15e9f-541d-4ac7-a4fe-bf2496e45773" podUID="ae85070a-1d2a-48cd-a14a-e3fbf927c61f" Jul 14 19:08:21.222534 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:21.222504 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbf15e9f-541d-4ac7-a4fe-bf2496e45773" path="/var/lib/kubelet/pods/bbf15e9f-541d-4ac7-a4fe-bf2496e45773/volumes" Jul 14 19:08:21.336564 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:21.336530 2573 generic.go:356] "Generic (PLEG): container finished" podID="ae85070a-1d2a-48cd-a14a-e3fbf927c61f" containerID="927ad21282918fa3055cc089d647bd53ce8831f7cf28b9edff64253dda7559c4" exitCode=0 Jul 14 19:08:21.336751 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:21.336569 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7944-139954953695280" event={"ID":"ae85070a-1d2a-48cd-a14a-e3fbf927c61f","Type":"ContainerDied","Data":"927ad21282918fa3055cc089d647bd53ce8831f7cf28b9edff64253dda7559c4"} Jul 14 19:08:22.501803 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.501780 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953695280" Jul 14 19:08:22.526246 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.526211 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7944-139954953695280_ae85070a-1d2a-48cd-a14a-e3fbf927c61f/test-curl-7944-139954953695280/0.log" Jul 14 19:08:22.556313 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.556273 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7944-139954953695280"] Jul 14 19:08:22.562145 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.562125 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7944-139954953695280"] Jul 14 19:08:22.597874 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.597822 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxr9l\" (UniqueName: \"kubernetes.io/projected/ae85070a-1d2a-48cd-a14a-e3fbf927c61f-kube-api-access-sxr9l\") pod \"ae85070a-1d2a-48cd-a14a-e3fbf927c61f\" (UID: \"ae85070a-1d2a-48cd-a14a-e3fbf927c61f\") " Jul 14 19:08:22.599983 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.599960 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae85070a-1d2a-48cd-a14a-e3fbf927c61f-kube-api-access-sxr9l" (OuterVolumeSpecName: "kube-api-access-sxr9l") pod "ae85070a-1d2a-48cd-a14a-e3fbf927c61f" (UID: "ae85070a-1d2a-48cd-a14a-e3fbf927c61f"). InnerVolumeSpecName "kube-api-access-sxr9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:08:22.693253 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.693226 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7944-139954953696288"] Jul 14 19:08:22.693729 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.693716 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae85070a-1d2a-48cd-a14a-e3fbf927c61f" containerName="test-curl-7944-139954953695280" Jul 14 19:08:22.693780 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.693732 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae85070a-1d2a-48cd-a14a-e3fbf927c61f" containerName="test-curl-7944-139954953695280" Jul 14 19:08:22.693821 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.693816 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae85070a-1d2a-48cd-a14a-e3fbf927c61f" containerName="test-curl-7944-139954953695280" Jul 14 19:08:22.697034 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.697020 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:22.699192 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.699154 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxr9l\" (UniqueName: \"kubernetes.io/projected/ae85070a-1d2a-48cd-a14a-e3fbf927c61f-kube-api-access-sxr9l\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:08:22.701775 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.701752 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7944-139954953696288"] Jul 14 19:08:22.800269 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.800241 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8twt\" (UniqueName: \"kubernetes.io/projected/0b8bbada-5069-4f7d-b826-cfc5b50e6d59-kube-api-access-v8twt\") pod \"test-curl-7944-139954953696288\" (UID: \"0b8bbada-5069-4f7d-b826-cfc5b50e6d59\") " pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:22.901718 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.901650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8twt\" (UniqueName: \"kubernetes.io/projected/0b8bbada-5069-4f7d-b826-cfc5b50e6d59-kube-api-access-v8twt\") pod \"test-curl-7944-139954953696288\" (UID: \"0b8bbada-5069-4f7d-b826-cfc5b50e6d59\") " pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:22.910617 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:22.910590 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8twt\" (UniqueName: \"kubernetes.io/projected/0b8bbada-5069-4f7d-b826-cfc5b50e6d59-kube-api-access-v8twt\") pod \"test-curl-7944-139954953696288\" (UID: \"0b8bbada-5069-4f7d-b826-cfc5b50e6d59\") " pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:23.008130 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:23.008105 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:23.170763 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:23.170738 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7944-139954953696288"] Jul 14 19:08:23.171796 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:08:23.171769 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b8bbada_5069_4f7d_b826_cfc5b50e6d59.slice/crio-a511ef4b823bdee3908cc3ce5d363fcf06d405007f477e2b4403b4a4b416a4a6 WatchSource:0}: Error finding container a511ef4b823bdee3908cc3ce5d363fcf06d405007f477e2b4403b4a4b416a4a6: Status 404 returned error can't find the container with id a511ef4b823bdee3908cc3ce5d363fcf06d405007f477e2b4403b4a4b416a4a6 Jul 14 19:08:23.227608 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:23.227580 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae85070a-1d2a-48cd-a14a-e3fbf927c61f" path="/var/lib/kubelet/pods/ae85070a-1d2a-48cd-a14a-e3fbf927c61f/volumes" Jul 14 19:08:23.351334 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:23.351308 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953695280" Jul 14 19:08:23.351464 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:23.351336 2573 scope.go:117] "RemoveContainer" containerID="927ad21282918fa3055cc089d647bd53ce8831f7cf28b9edff64253dda7559c4" Jul 14 19:08:23.352715 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:23.352694 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7944-139954953696288" event={"ID":"0b8bbada-5069-4f7d-b826-cfc5b50e6d59","Type":"ContainerStarted","Data":"a511ef4b823bdee3908cc3ce5d363fcf06d405007f477e2b4403b4a4b416a4a6"} Jul 14 19:08:24.359726 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:24.359685 2573 generic.go:356] "Generic (PLEG): container finished" podID="0b8bbada-5069-4f7d-b826-cfc5b50e6d59" containerID="57b73b0f3448f56df1c72f2b2147db47a548e63f4b8f72dc00f386ff0ce681a7" exitCode=0 Jul 14 19:08:24.360206 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:24.359767 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7944-139954953696288" event={"ID":"0b8bbada-5069-4f7d-b826-cfc5b50e6d59","Type":"ContainerDied","Data":"57b73b0f3448f56df1c72f2b2147db47a548e63f4b8f72dc00f386ff0ce681a7"} Jul 14 19:08:25.518010 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:25.517986 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:25.543593 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:25.543574 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7944-139954953696288_0b8bbada-5069-4f7d-b826-cfc5b50e6d59/test-curl-7944-139954953696288/0.log" Jul 14 19:08:25.574497 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:25.574469 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7944-139954953696288"] Jul 14 19:08:25.578839 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:25.578816 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7944-139954953696288"] Jul 14 19:08:25.630042 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:25.629983 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8twt\" (UniqueName: \"kubernetes.io/projected/0b8bbada-5069-4f7d-b826-cfc5b50e6d59-kube-api-access-v8twt\") pod \"0b8bbada-5069-4f7d-b826-cfc5b50e6d59\" (UID: \"0b8bbada-5069-4f7d-b826-cfc5b50e6d59\") " Jul 14 19:08:25.631936 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:25.631907 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8bbada-5069-4f7d-b826-cfc5b50e6d59-kube-api-access-v8twt" (OuterVolumeSpecName: "kube-api-access-v8twt") pod "0b8bbada-5069-4f7d-b826-cfc5b50e6d59" (UID: "0b8bbada-5069-4f7d-b826-cfc5b50e6d59"). InnerVolumeSpecName "kube-api-access-v8twt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:08:25.731417 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:25.731387 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8twt\" (UniqueName: \"kubernetes.io/projected/0b8bbada-5069-4f7d-b826-cfc5b50e6d59-kube-api-access-v8twt\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:08:26.374309 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:26.374258 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:26.374477 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:26.374323 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a511ef4b823bdee3908cc3ce5d363fcf06d405007f477e2b4403b4a4b416a4a6" Jul 14 19:08:26.385982 ip-10-0-132-224 kubenswrapper[2573]: E0714 19:08:26.385957 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7944-139954953696288\" is forbidden: User \"system:node:ip-10-0-132-224.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-132-224.ec2.internal' and this object" podUID="0b8bbada-5069-4f7d-b826-cfc5b50e6d59" pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:27.223519 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:27.223486 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b8bbada-5069-4f7d-b826-cfc5b50e6d59" path="/var/lib/kubelet/pods/0b8bbada-5069-4f7d-b826-cfc5b50e6d59/volumes" Jul 14 19:08:29.127380 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:29.127350 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-52ef23-l428d"] Jul 14 19:08:29.127899 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:29.127883 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b8bbada-5069-4f7d-b826-cfc5b50e6d59" containerName="test-curl-7944-139954953696288" Jul 14 19:08:29.127899 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:29.127899 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b8bbada-5069-4f7d-b826-cfc5b50e6d59" containerName="test-curl-7944-139954953696288" Jul 14 19:08:29.128025 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:29.128005 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b8bbada-5069-4f7d-b826-cfc5b50e6d59" containerName="test-curl-7944-139954953696288" Jul 14 19:08:29.131534 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:29.131512 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-52ef23-l428d" Jul 14 19:08:29.135140 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:29.135120 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 19:08:29.140180 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:29.140132 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-52ef23-l428d"] Jul 14 19:08:29.263533 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:29.263502 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e58cec1-ec33-4228-91cb-e6f8af973b4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-52ef23-l428d\" (UID: \"2e58cec1-ec33-4228-91cb-e6f8af973b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-52ef23-l428d" Jul 14 19:08:29.365002 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:29.364962 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e58cec1-ec33-4228-91cb-e6f8af973b4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-52ef23-l428d\" (UID: \"2e58cec1-ec33-4228-91cb-e6f8af973b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-52ef23-l428d" Jul 14 19:08:29.365584 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:29.365563 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e58cec1-ec33-4228-91cb-e6f8af973b4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-52ef23-l428d\" (UID: \"2e58cec1-ec33-4228-91cb-e6f8af973b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-52ef23-l428d" Jul 14 19:08:29.445303 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:29.445261 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-52ef23-l428d" Jul 14 19:08:29.588935 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:29.588906 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-52ef23-l428d"] Jul 14 19:08:29.589849 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:08:29.589823 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e58cec1_ec33_4228_91cb_e6f8af973b4c.slice/crio-66a99055a11e55cb5d91aace4fc0580099a48e24fdb745d9a32f0f77f05416e0 WatchSource:0}: Error finding container 66a99055a11e55cb5d91aace4fc0580099a48e24fdb745d9a32f0f77f05416e0: Status 404 returned error can't find the container with id 66a99055a11e55cb5d91aace4fc0580099a48e24fdb745d9a32f0f77f05416e0 Jul 14 19:08:29.766700 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:29.766598 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:08:29.864857 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:29.864829 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:08:30.399809 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:30.399780 2573 generic.go:356] "Generic (PLEG): container finished" podID="2e58cec1-ec33-4228-91cb-e6f8af973b4c" containerID="e4986cb36b9c3e1d13bb1bf06a860dc8bf095081b5861dbf839a458536527fa2" exitCode=0 Jul 14 19:08:30.400118 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:30.399866 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-52ef23-l428d" event={"ID":"2e58cec1-ec33-4228-91cb-e6f8af973b4c","Type":"ContainerDied","Data":"e4986cb36b9c3e1d13bb1bf06a860dc8bf095081b5861dbf839a458536527fa2"} Jul 14 19:08:30.400118 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:30.399927 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-52ef23-l428d" event={"ID":"2e58cec1-ec33-4228-91cb-e6f8af973b4c","Type":"ContainerStarted","Data":"66a99055a11e55cb5d91aace4fc0580099a48e24fdb745d9a32f0f77f05416e0"} Jul 14 19:08:31.550821 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:31.550797 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-52ef23-l428d" Jul 14 19:08:31.689082 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:31.689059 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e58cec1-ec33-4228-91cb-e6f8af973b4c-maas-api-service-ca\") pod \"2e58cec1-ec33-4228-91cb-e6f8af973b4c\" (UID: \"2e58cec1-ec33-4228-91cb-e6f8af973b4c\") " Jul 14 19:08:31.689384 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:31.689364 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e58cec1-ec33-4228-91cb-e6f8af973b4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e58cec1-ec33-4228-91cb-e6f8af973b4c" (UID: "2e58cec1-ec33-4228-91cb-e6f8af973b4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:08:31.790391 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:31.790360 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e58cec1-ec33-4228-91cb-e6f8af973b4c-maas-api-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:08:32.413583 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:32.413553 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-52ef23-l428d" Jul 14 19:08:32.413766 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:32.413587 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-52ef23-l428d" event={"ID":"2e58cec1-ec33-4228-91cb-e6f8af973b4c","Type":"ContainerDied","Data":"66a99055a11e55cb5d91aace4fc0580099a48e24fdb745d9a32f0f77f05416e0"} Jul 14 19:08:32.413766 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:32.413638 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66a99055a11e55cb5d91aace4fc0580099a48e24fdb745d9a32f0f77f05416e0" Jul 14 19:08:39.675049 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:39.674969 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4"] Jul 14 19:08:39.675503 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:39.675261 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" podUID="6c215e7c-39ec-4a65-aeab-a67c2752c2f9" containerName="maas-api" containerID="cri-o://2c1d46087c5379ff3b17991f959a6d0b8d4594766c88b1f01286acf708abb3fc" gracePeriod=30 Jul 14 19:08:39.709731 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:39.709699 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:08:39.951019 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:39.950996 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:08:40.071554 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.071520 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkt87\" (UniqueName: \"kubernetes.io/projected/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-kube-api-access-qkt87\") pod \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\" (UID: \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\") " Jul 14 19:08:40.071745 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.071575 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-openshift-service-ca\") pod \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\" (UID: \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\") " Jul 14 19:08:40.071745 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.071639 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-maas-api-tls\") pod \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\" (UID: \"6c215e7c-39ec-4a65-aeab-a67c2752c2f9\") " Jul 14 19:08:40.071985 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.071955 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c215e7c-39ec-4a65-aeab-a67c2752c2f9" (UID: "6c215e7c-39ec-4a65-aeab-a67c2752c2f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:08:40.073755 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.073726 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-kube-api-access-qkt87" (OuterVolumeSpecName: "kube-api-access-qkt87") pod "6c215e7c-39ec-4a65-aeab-a67c2752c2f9" (UID: "6c215e7c-39ec-4a65-aeab-a67c2752c2f9"). InnerVolumeSpecName "kube-api-access-qkt87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:08:40.073868 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.073796 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c215e7c-39ec-4a65-aeab-a67c2752c2f9" (UID: "6c215e7c-39ec-4a65-aeab-a67c2752c2f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:08:40.173159 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.173130 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-maas-api-tls\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:08:40.173159 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.173154 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkt87\" (UniqueName: \"kubernetes.io/projected/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-kube-api-access-qkt87\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:08:40.173159 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.173163 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c215e7c-39ec-4a65-aeab-a67c2752c2f9-openshift-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:08:40.463889 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.463850 2573 generic.go:356] "Generic (PLEG): container finished" podID="6c215e7c-39ec-4a65-aeab-a67c2752c2f9" containerID="2c1d46087c5379ff3b17991f959a6d0b8d4594766c88b1f01286acf708abb3fc" exitCode=0 Jul 14 19:08:40.464038 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.463907 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" Jul 14 19:08:40.464038 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.463934 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" event={"ID":"6c215e7c-39ec-4a65-aeab-a67c2752c2f9","Type":"ContainerDied","Data":"2c1d46087c5379ff3b17991f959a6d0b8d4594766c88b1f01286acf708abb3fc"} Jul 14 19:08:40.464038 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.464023 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4" event={"ID":"6c215e7c-39ec-4a65-aeab-a67c2752c2f9","Type":"ContainerDied","Data":"ae9c6b31163abf6db759c49346fe07237b291fb7b3b95bd70060d82684249f78"} Jul 14 19:08:40.464165 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.464052 2573 scope.go:117] "RemoveContainer" containerID="2c1d46087c5379ff3b17991f959a6d0b8d4594766c88b1f01286acf708abb3fc" Jul 14 19:08:40.474116 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.474094 2573 scope.go:117] "RemoveContainer" containerID="2c1d46087c5379ff3b17991f959a6d0b8d4594766c88b1f01286acf708abb3fc" Jul 14 19:08:40.474419 ip-10-0-132-224 kubenswrapper[2573]: E0714 19:08:40.474401 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c1d46087c5379ff3b17991f959a6d0b8d4594766c88b1f01286acf708abb3fc\": container with ID starting with 2c1d46087c5379ff3b17991f959a6d0b8d4594766c88b1f01286acf708abb3fc not found: ID does not exist" containerID="2c1d46087c5379ff3b17991f959a6d0b8d4594766c88b1f01286acf708abb3fc" Jul 14 19:08:40.474474 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.474428 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c1d46087c5379ff3b17991f959a6d0b8d4594766c88b1f01286acf708abb3fc"} err="failed to get container status \"2c1d46087c5379ff3b17991f959a6d0b8d4594766c88b1f01286acf708abb3fc\": rpc error: code = NotFound desc = could not find container \"2c1d46087c5379ff3b17991f959a6d0b8d4594766c88b1f01286acf708abb3fc\": container with ID starting with 2c1d46087c5379ff3b17991f959a6d0b8d4594766c88b1f01286acf708abb3fc not found: ID does not exist" Jul 14 19:08:40.489024 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.489002 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4"] Jul 14 19:08:40.491612 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:40.491594 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-52ef23-75b4dbff55-z4cx4"] Jul 14 19:08:41.222418 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:41.222383 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c215e7c-39ec-4a65-aeab-a67c2752c2f9" path="/var/lib/kubelet/pods/6c215e7c-39ec-4a65-aeab-a67c2752c2f9/volumes" Jul 14 19:08:42.821035 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:42.820995 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:08:42.891328 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:42.891298 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:08:44.299631 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:44.299591 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:08:50.936439 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:50.936405 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:08:51.032585 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:51.032548 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:08:55.014526 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.014491 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:08:55.265464 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.265379 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:08:55.391021 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.390979 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn"] Jul 14 19:08:55.391576 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.391557 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e58cec1-ec33-4228-91cb-e6f8af973b4c" containerName="revoke-keys" Jul 14 19:08:55.391576 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.391578 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e58cec1-ec33-4228-91cb-e6f8af973b4c" containerName="revoke-keys" Jul 14 19:08:55.391750 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.391624 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c215e7c-39ec-4a65-aeab-a67c2752c2f9" containerName="maas-api" Jul 14 19:08:55.391750 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.391632 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c215e7c-39ec-4a65-aeab-a67c2752c2f9" containerName="maas-api" Jul 14 19:08:55.391856 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.391759 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c215e7c-39ec-4a65-aeab-a67c2752c2f9" containerName="maas-api" Jul 14 19:08:55.391856 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.391781 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e58cec1-ec33-4228-91cb-e6f8af973b4c" containerName="revoke-keys" Jul 14 19:08:55.395339 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.395320 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn" Jul 14 19:08:55.398148 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.398132 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 19:08:55.401828 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.401803 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6728abe0-7ffd-4320-9a30-8d8f5a6de35e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn\" (UID: \"6728abe0-7ffd-4320-9a30-8d8f5a6de35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn" Jul 14 19:08:55.404090 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.404066 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn"] Jul 14 19:08:55.502833 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.502797 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6728abe0-7ffd-4320-9a30-8d8f5a6de35e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn\" (UID: \"6728abe0-7ffd-4320-9a30-8d8f5a6de35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn" Jul 14 19:08:55.503508 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.503486 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6728abe0-7ffd-4320-9a30-8d8f5a6de35e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn\" (UID: \"6728abe0-7ffd-4320-9a30-8d8f5a6de35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn" Jul 14 19:08:55.708422 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.708392 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn" Jul 14 19:08:55.853591 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.853566 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn"] Jul 14 19:08:55.855255 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:08:55.855226 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6728abe0_7ffd_4320_9a30_8d8f5a6de35e.slice/crio-bc38ccb64f1a97196bfec3a07be14fb36e5dfdfce5d5ec3fa3c012acf4f4a77f WatchSource:0}: Error finding container bc38ccb64f1a97196bfec3a07be14fb36e5dfdfce5d5ec3fa3c012acf4f4a77f: Status 404 returned error can't find the container with id bc38ccb64f1a97196bfec3a07be14fb36e5dfdfce5d5ec3fa3c012acf4f4a77f Jul 14 19:08:55.923082 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:55.923055 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:08:56.023785 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:56.023750 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:08:56.569373 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:56.569339 2573 generic.go:356] "Generic (PLEG): container finished" podID="6728abe0-7ffd-4320-9a30-8d8f5a6de35e" containerID="49f10036bbcdd3efa6544e0a3dad771b22ff86d045d618c06a27b8ef4f209e6a" exitCode=0 Jul 14 19:08:56.569551 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:56.569427 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn" event={"ID":"6728abe0-7ffd-4320-9a30-8d8f5a6de35e","Type":"ContainerDied","Data":"49f10036bbcdd3efa6544e0a3dad771b22ff86d045d618c06a27b8ef4f209e6a"} Jul 14 19:08:56.569551 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:56.569478 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn" event={"ID":"6728abe0-7ffd-4320-9a30-8d8f5a6de35e","Type":"ContainerStarted","Data":"bc38ccb64f1a97196bfec3a07be14fb36e5dfdfce5d5ec3fa3c012acf4f4a77f"} Jul 14 19:08:57.720694 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:57.720667 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn" Jul 14 19:08:57.825385 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:57.825356 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6728abe0-7ffd-4320-9a30-8d8f5a6de35e-maas-api-service-ca\") pod \"6728abe0-7ffd-4320-9a30-8d8f5a6de35e\" (UID: \"6728abe0-7ffd-4320-9a30-8d8f5a6de35e\") " Jul 14 19:08:57.825713 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:57.825688 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6728abe0-7ffd-4320-9a30-8d8f5a6de35e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6728abe0-7ffd-4320-9a30-8d8f5a6de35e" (UID: "6728abe0-7ffd-4320-9a30-8d8f5a6de35e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:08:57.926353 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:57.926263 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6728abe0-7ffd-4320-9a30-8d8f5a6de35e-maas-api-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:08:58.583336 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:58.583302 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn" event={"ID":"6728abe0-7ffd-4320-9a30-8d8f5a6de35e","Type":"ContainerDied","Data":"bc38ccb64f1a97196bfec3a07be14fb36e5dfdfce5d5ec3fa3c012acf4f4a77f"} Jul 14 19:08:58.583524 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:58.583342 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn" Jul 14 19:08:58.583524 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:08:58.583359 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc38ccb64f1a97196bfec3a07be14fb36e5dfdfce5d5ec3fa3c012acf4f4a77f" Jul 14 19:09:06.019440 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.019405 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt"] Jul 14 19:09:06.019937 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.019686 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" podUID="da363f37-749c-409a-adef-710c875563ff" containerName="maas-api" containerID="cri-o://13568f88ffc6a864d275b2a4f46ed4cf14a78047ccacfb78eb47a4835d0c88de" gracePeriod=30 Jul 14 19:09:06.052243 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.052219 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:09:06.283860 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.283806 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" Jul 14 19:09:06.408566 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.408540 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da363f37-749c-409a-adef-710c875563ff-openshift-service-ca\") pod \"da363f37-749c-409a-adef-710c875563ff\" (UID: \"da363f37-749c-409a-adef-710c875563ff\") " Jul 14 19:09:06.408750 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.408574 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r7lm\" (UniqueName: \"kubernetes.io/projected/da363f37-749c-409a-adef-710c875563ff-kube-api-access-4r7lm\") pod \"da363f37-749c-409a-adef-710c875563ff\" (UID: \"da363f37-749c-409a-adef-710c875563ff\") " Jul 14 19:09:06.408750 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.408714 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da363f37-749c-409a-adef-710c875563ff-maas-api-tls\") pod \"da363f37-749c-409a-adef-710c875563ff\" (UID: \"da363f37-749c-409a-adef-710c875563ff\") " Jul 14 19:09:06.408928 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.408899 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da363f37-749c-409a-adef-710c875563ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "da363f37-749c-409a-adef-710c875563ff" (UID: "da363f37-749c-409a-adef-710c875563ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:09:06.409056 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.409040 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da363f37-749c-409a-adef-710c875563ff-openshift-service-ca\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:09:06.410658 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.410632 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da363f37-749c-409a-adef-710c875563ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "da363f37-749c-409a-adef-710c875563ff" (UID: "da363f37-749c-409a-adef-710c875563ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:09:06.410783 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.410720 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da363f37-749c-409a-adef-710c875563ff-kube-api-access-4r7lm" (OuterVolumeSpecName: "kube-api-access-4r7lm") pod "da363f37-749c-409a-adef-710c875563ff" (UID: "da363f37-749c-409a-adef-710c875563ff"). InnerVolumeSpecName "kube-api-access-4r7lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:09:06.509807 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.509784 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da363f37-749c-409a-adef-710c875563ff-maas-api-tls\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:09:06.509807 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.509808 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r7lm\" (UniqueName: \"kubernetes.io/projected/da363f37-749c-409a-adef-710c875563ff-kube-api-access-4r7lm\") on node \"ip-10-0-132-224.ec2.internal\" DevicePath \"\"" Jul 14 19:09:06.635160 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.635094 2573 generic.go:356] "Generic (PLEG): container finished" podID="da363f37-749c-409a-adef-710c875563ff" containerID="13568f88ffc6a864d275b2a4f46ed4cf14a78047ccacfb78eb47a4835d0c88de" exitCode=0 Jul 14 19:09:06.635160 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.635135 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" Jul 14 19:09:06.635339 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.635188 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" event={"ID":"da363f37-749c-409a-adef-710c875563ff","Type":"ContainerDied","Data":"13568f88ffc6a864d275b2a4f46ed4cf14a78047ccacfb78eb47a4835d0c88de"} Jul 14 19:09:06.635339 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.635242 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" event={"ID":"da363f37-749c-409a-adef-710c875563ff","Type":"ContainerDied","Data":"eb8c1831c1b17a63cbe03451f6ef022215ea426898422ec02fb93674d1421a4d"} Jul 14 19:09:06.635339 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.635267 2573 scope.go:117] "RemoveContainer" containerID="13568f88ffc6a864d275b2a4f46ed4cf14a78047ccacfb78eb47a4835d0c88de" Jul 14 19:09:06.645342 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.645322 2573 scope.go:117] "RemoveContainer" containerID="13568f88ffc6a864d275b2a4f46ed4cf14a78047ccacfb78eb47a4835d0c88de" Jul 14 19:09:06.645598 ip-10-0-132-224 kubenswrapper[2573]: E0714 19:09:06.645579 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13568f88ffc6a864d275b2a4f46ed4cf14a78047ccacfb78eb47a4835d0c88de\": container with ID starting with 13568f88ffc6a864d275b2a4f46ed4cf14a78047ccacfb78eb47a4835d0c88de not found: ID does not exist" containerID="13568f88ffc6a864d275b2a4f46ed4cf14a78047ccacfb78eb47a4835d0c88de" Jul 14 19:09:06.645651 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.645608 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13568f88ffc6a864d275b2a4f46ed4cf14a78047ccacfb78eb47a4835d0c88de"} err="failed to get container status \"13568f88ffc6a864d275b2a4f46ed4cf14a78047ccacfb78eb47a4835d0c88de\": rpc error: code = NotFound desc = could not find container \"13568f88ffc6a864d275b2a4f46ed4cf14a78047ccacfb78eb47a4835d0c88de\": container with ID starting with 13568f88ffc6a864d275b2a4f46ed4cf14a78047ccacfb78eb47a4835d0c88de not found: ID does not exist" Jul 14 19:09:06.658712 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.658691 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt"] Jul 14 19:09:06.662722 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:06.662704 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt"] Jul 14 19:09:07.222707 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:07.222676 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da363f37-749c-409a-adef-710c875563ff" path="/var/lib/kubelet/pods/da363f37-749c-409a-adef-710c875563ff/volumes" Jul 14 19:09:09.203306 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:09.203258 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-10fa01-57c5545444-lpjgt" podUID="da363f37-749c-409a-adef-710c875563ff" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.78:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 14 19:09:10.480714 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:10.480675 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:09:17.128219 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:17.128157 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:09:17.207333 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:17.207277 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:09:21.236980 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:21.236934 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:09:21.473937 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:21.473897 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:09:27.231741 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:27.231707 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-46m4j_8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d/manager/0.log" Jul 14 19:09:27.368926 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:27.368894 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-knkxx_fcce408f-a62a-426c-afce-01250b0490ec/manager/0.log" Jul 14 19:09:27.512927 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:27.512850 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-846475b4dc-cj4kt_ddf090f3-ed96-418b-85ef-9f6f25f387e6/manager/0.log" Jul 14 19:09:27.638832 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:27.638802 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-hxhf5_b9077472-e6b7-4938-a5c1-6acf578d0e13/server/0.log" Jul 14 19:09:27.781183 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:27.781103 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fgbsf_460ba81a-42b2-48e0-9fb0-ac035c67fd36/manager/0.log" Jul 14 19:09:28.049257 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:28.049175 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d948cf9bc-5vfql_45be84e1-0678-4cb0-aaf1-8898001e3eeb/copy-manifests/0.log" Jul 14 19:09:28.068779 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:28.068749 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d948cf9bc-5vfql_45be84e1-0678-4cb0-aaf1-8898001e3eeb/manager/0.log" Jul 14 19:09:28.589200 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:28.589177 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-587fdf675b-jg8sx_87991a28-bafc-431a-9242-1982dc7874ab/maas-api/0.log" Jul 14 19:09:28.719090 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:28.719059 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734245-r5vrd_c1155c96-7e2e-4e90-9222-5cc5517a4708/cleanup/0.log" Jul 14 19:09:28.848932 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:28.848865 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734260-kq6xr_fd212eb6-7b43-4b30-85c8-57d1dfe4078c/cleanup/0.log" Jul 14 19:09:29.253926 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:29.253897 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-82771c-j5rjm_c4c864f7-e7c5-4206-8c3c-fd1efdc15056/revoke-keys/0.log" Jul 14 19:09:29.401837 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:29.401804 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-601221-7hqqx_1aa215b6-f4b1-4333-a8fa-7585b911cb22/revoke-keys/0.log" Jul 14 19:09:29.660231 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:29.660155 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-0895b5-gkr6j_1ff9bca6-c23e-40ef-840f-14b668eb79a9/revoke-keys/0.log" Jul 14 19:09:29.820058 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:29.820030 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-08413e-m4dhk_abb6bafe-ed56-435a-9096-6a297379f301/revoke-keys/0.log" Jul 14 19:09:30.305102 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:30.305072 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-52ef23-l428d_2e58cec1-ec33-4228-91cb-e6f8af973b4c/revoke-keys/0.log" Jul 14 19:09:30.424586 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:30.424555 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-10fa01-nk9qn_6728abe0-7ffd-4320-9a30-8d8f5a6de35e/revoke-keys/0.log" Jul 14 19:09:30.539994 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:30.539967 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-hwt49_2fdcd8d1-06ca-42f9-a8c8-14d646614ec4/postgres/0.log" Jul 14 19:09:32.081670 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:32.081627 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-tjswq_3eeaba31-888b-46a4-9c65-38f28e914254/manager/0.log" Jul 14 19:09:32.200932 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:32.200903 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-s95x5_73919d62-68e8-4b94-9508-5495e52a1902/manager/0.log" Jul 14 19:09:32.454049 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:32.454013 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jhn8w_7a5e68e3-6771-4f17-bb6b-88d2d5f0460f/registry-server/0.log" Jul 14 19:09:32.846995 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:32.846926 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-p2j9n_fb25b1f0-68d0-410e-9959-66757fffd214/manager/0.log" Jul 14 19:09:33.237679 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:33.237655 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4tgcnj_99b9b6de-ac61-470e-948e-e48e88cf4039/istio-proxy/0.log" Jul 14 19:09:33.629081 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:33.628997 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d44f677bf-zv7jq_b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c/kube-auth-proxy/0.log" Jul 14 19:09:33.765095 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:33.765059 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rlbm8_83981a26-6460-417f-9442-602373954802/istio-proxy/1.log" Jul 14 19:09:34.037774 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:34.037741 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-k8tcp_64f2c200-c428-4c62-ad4c-d8f13482a641/payload-processing/0.log" Jul 14 19:09:34.164999 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:34.164967 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59f484b746-8hs6t_c7cf164f-807a-4191-acdd-e7d5f3c6de75/router/0.log" Jul 14 19:09:34.661819 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:34.661787 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-gpk6z_0e80dcfd-345e-49fb-be55-a3f8770a9903/main/0.log" Jul 14 19:09:34.796782 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:34.796753 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-j869s_5f757be3-78b0-4951-9147-6f8102a02cea/main/0.log" Jul 14 19:09:39.076983 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.076949 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-26nj6/must-gather-bndm5"] Jul 14 19:09:39.077459 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.077445 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6728abe0-7ffd-4320-9a30-8d8f5a6de35e" containerName="revoke-keys" Jul 14 19:09:39.077459 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.077460 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6728abe0-7ffd-4320-9a30-8d8f5a6de35e" containerName="revoke-keys" Jul 14 19:09:39.077536 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.077483 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da363f37-749c-409a-adef-710c875563ff" containerName="maas-api" Jul 14 19:09:39.077536 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.077488 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="da363f37-749c-409a-adef-710c875563ff" containerName="maas-api" Jul 14 19:09:39.077603 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.077560 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6728abe0-7ffd-4320-9a30-8d8f5a6de35e" containerName="revoke-keys" Jul 14 19:09:39.077603 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.077571 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="da363f37-749c-409a-adef-710c875563ff" containerName="maas-api" Jul 14 19:09:39.081258 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.081238 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26nj6/must-gather-bndm5" Jul 14 19:09:39.084175 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.084150 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26nj6\"/\"openshift-service-ca.crt\"" Jul 14 19:09:39.084175 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.084150 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26nj6\"/\"kube-root-ca.crt\"" Jul 14 19:09:39.085444 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.085428 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-26nj6\"/\"default-dockercfg-9br9n\"" Jul 14 19:09:39.091642 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.091620 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26nj6/must-gather-bndm5"] Jul 14 19:09:39.096024 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.095992 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:09:39.126973 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.126949 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fb181287-1d24-4547-be73-78f823b5f28b-must-gather-output\") pod \"must-gather-bndm5\" (UID: \"fb181287-1d24-4547-be73-78f823b5f28b\") " pod="openshift-must-gather-26nj6/must-gather-bndm5" Jul 14 19:09:39.127087 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.127065 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2wtl\" (UniqueName: \"kubernetes.io/projected/fb181287-1d24-4547-be73-78f823b5f28b-kube-api-access-p2wtl\") pod \"must-gather-bndm5\" (UID: \"fb181287-1d24-4547-be73-78f823b5f28b\") " pod="openshift-must-gather-26nj6/must-gather-bndm5" Jul 14 19:09:39.194828 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.194800 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:09:39.228449 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.228426 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fb181287-1d24-4547-be73-78f823b5f28b-must-gather-output\") pod \"must-gather-bndm5\" (UID: \"fb181287-1d24-4547-be73-78f823b5f28b\") " pod="openshift-must-gather-26nj6/must-gather-bndm5" Jul 14 19:09:39.228562 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.228531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2wtl\" (UniqueName: \"kubernetes.io/projected/fb181287-1d24-4547-be73-78f823b5f28b-kube-api-access-p2wtl\") pod \"must-gather-bndm5\" (UID: \"fb181287-1d24-4547-be73-78f823b5f28b\") " pod="openshift-must-gather-26nj6/must-gather-bndm5" Jul 14 19:09:39.228766 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.228751 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fb181287-1d24-4547-be73-78f823b5f28b-must-gather-output\") pod \"must-gather-bndm5\" (UID: \"fb181287-1d24-4547-be73-78f823b5f28b\") " pod="openshift-must-gather-26nj6/must-gather-bndm5" Jul 14 19:09:39.243854 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.243832 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2wtl\" (UniqueName: \"kubernetes.io/projected/fb181287-1d24-4547-be73-78f823b5f28b-kube-api-access-p2wtl\") pod \"must-gather-bndm5\" (UID: \"fb181287-1d24-4547-be73-78f823b5f28b\") " pod="openshift-must-gather-26nj6/must-gather-bndm5" Jul 14 19:09:39.391605 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.391580 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26nj6/must-gather-bndm5" Jul 14 19:09:39.545269 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.545242 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26nj6/must-gather-bndm5"] Jul 14 19:09:39.546529 ip-10-0-132-224 kubenswrapper[2573]: W0714 19:09:39.546504 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb181287_1d24_4547_be73_78f823b5f28b.slice/crio-3ac05b2c4bc26978d1c243a25d335f838e9a807c3f2ba69a5e1f439c509c9961 WatchSource:0}: Error finding container 3ac05b2c4bc26978d1c243a25d335f838e9a807c3f2ba69a5e1f439c509c9961: Status 404 returned error can't find the container with id 3ac05b2c4bc26978d1c243a25d335f838e9a807c3f2ba69a5e1f439c509c9961 Jul 14 19:09:39.837207 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:39.837171 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26nj6/must-gather-bndm5" event={"ID":"fb181287-1d24-4547-be73-78f823b5f28b","Type":"ContainerStarted","Data":"3ac05b2c4bc26978d1c243a25d335f838e9a807c3f2ba69a5e1f439c509c9961"} Jul 14 19:09:40.848247 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:40.848216 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26nj6/must-gather-bndm5" event={"ID":"fb181287-1d24-4547-be73-78f823b5f28b","Type":"ContainerStarted","Data":"8295c84410c969df57b036dbf6cee2cf91f0e8c0f91a42ea494f2689af15772c"} Jul 14 19:09:41.857747 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:41.857713 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26nj6/must-gather-bndm5" event={"ID":"fb181287-1d24-4547-be73-78f823b5f28b","Type":"ContainerStarted","Data":"75b2dd60370337f6bf662010398e9d93ea1694dd90efd7cc55e35054af2005f6"} Jul 14 19:09:41.878779 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:41.878720 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-26nj6/must-gather-bndm5" podStartSLOduration=1.774573809 podStartE2EDuration="2.878697061s" totalImagesPullingTime="1.104123252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:09:41.87772208 +0000 UTC m=+2883.415169865" watchObservedRunningTime="2026-07-14 19:09:41.878697061 +0000 UTC m=+2883.416144845" Jul 14 19:09:42.354508 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:42.354473 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gm8z4_0049cf44-b6fe-4c7c-8e53-adbd70b7eb46/global-pull-secret-syncer/0.log" Jul 14 19:09:42.508188 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:42.508155 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" Jul 14 19:09:42.665710 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:42.665631 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-224.ec2.internal_13b1f4d964eb8d8619f16afcc6da67af/haproxy/0.log" Jul 14 19:09:48.642218 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:48.642184 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-tjswq_3eeaba31-888b-46a4-9c65-38f28e914254/manager/0.log" Jul 14 19:09:48.672841 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:48.672805 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-s95x5_73919d62-68e8-4b94-9508-5495e52a1902/manager/0.log" Jul 14 19:09:48.751148 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:48.751108 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jhn8w_7a5e68e3-6771-4f17-bb6b-88d2d5f0460f/registry-server/0.log" Jul 14 19:09:49.065405 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:49.065325 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-p2j9n_fb25b1f0-68d0-410e-9959-66757fffd214/manager/0.log" Jul 14 19:09:49.391297 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:49.391196 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:09:49.489975 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:49.489926 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-66x25_b6261fa8-ee48-45f8-9576-3a75f2effa71/konnectivity-agent/0.log" limit=1024 Jul 14 19:09:50.627250 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:50.627216 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5324143-4949-4048-aba8-6c9e979ab8dc/alertmanager/0.log" Jul 14 19:09:50.652913 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:50.652872 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5324143-4949-4048-aba8-6c9e979ab8dc/config-reloader/0.log" Jul 14 19:09:50.674453 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:50.674423 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5324143-4949-4048-aba8-6c9e979ab8dc/kube-rbac-proxy-web/0.log" Jul 14 19:09:50.698397 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:50.698367 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5324143-4949-4048-aba8-6c9e979ab8dc/kube-rbac-proxy/0.log" Jul 14 19:09:50.724531 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:50.724488 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5324143-4949-4048-aba8-6c9e979ab8dc/kube-rbac-proxy-metric/0.log" Jul 14 19:09:50.750424 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:50.750348 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5324143-4949-4048-aba8-6c9e979ab8dc/prom-label-proxy/0.log" Jul 14 19:09:50.777621 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:50.777594 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5324143-4949-4048-aba8-6c9e979ab8dc/init-config-reloader/0.log" Jul 14 19:09:50.830882 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:50.830854 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-l4fpb_24acf903-caba-43ed-bd46-cf61a79524c9/cluster-monitoring-operator/0.log" Jul 14 19:09:50.864026 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:50.864000 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-hbr4k_264c6200-8285-41b2-a645-5a025b3a7c6f/kube-state-metrics/0.log" Jul 14 19:09:50.884701 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:50.884678 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-hbr4k_264c6200-8285-41b2-a645-5a025b3a7c6f/kube-rbac-proxy-main/0.log" Jul 14 19:09:50.910169 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:50.910139 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-hbr4k_264c6200-8285-41b2-a645-5a025b3a7c6f/kube-rbac-proxy-self/0.log" Jul 14 19:09:50.974814 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:50.974783 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-z9kjd_9116c6f0-66e8-4503-8570-fd20f75816cb/monitoring-plugin/0.log" Jul 14 19:09:51.122459 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.122381 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nsfmd_64ba4ec1-e886-4191-b8ad-15606471581b/node-exporter/0.log" Jul 14 19:09:51.159048 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.159018 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nsfmd_64ba4ec1-e886-4191-b8ad-15606471581b/kube-rbac-proxy/0.log" Jul 14 19:09:51.186736 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.186704 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nsfmd_64ba4ec1-e886-4191-b8ad-15606471581b/init-textfile/0.log" Jul 14 19:09:51.298902 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.298866 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-q9ppd_07fada97-ddc6-4d47-b0c3-5d7816ea4027/kube-rbac-proxy-main/0.log" Jul 14 19:09:51.337545 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.337513 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-q9ppd_07fada97-ddc6-4d47-b0c3-5d7816ea4027/kube-rbac-proxy-self/0.log" Jul 14 19:09:51.393319 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.393224 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-q9ppd_07fada97-ddc6-4d47-b0c3-5d7816ea4027/openshift-state-metrics/0.log" Jul 14 19:09:51.667387 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.667275 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-rxmhj_07d20ec4-fefa-4651-ab31-5007b4c91d53/prometheus-operator-admission-webhook/0.log" Jul 14 19:09:51.699087 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.699059 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b665d5f69-xbh9c_0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb/telemeter-client/0.log" Jul 14 19:09:51.726153 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.726124 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b665d5f69-xbh9c_0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb/reload/0.log" Jul 14 19:09:51.756226 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.756191 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b665d5f69-xbh9c_0b11cad7-ebf8-44ed-a0db-bd6b36b5e2cb/kube-rbac-proxy/0.log" Jul 14 19:09:51.793676 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.793651 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797996db74-bzk2c_8cbd1903-2698-4c5d-8e73-2910b6991f36/thanos-query/0.log" Jul 14 19:09:51.816812 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.816782 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797996db74-bzk2c_8cbd1903-2698-4c5d-8e73-2910b6991f36/kube-rbac-proxy-web/0.log" Jul 14 19:09:51.838732 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.838701 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797996db74-bzk2c_8cbd1903-2698-4c5d-8e73-2910b6991f36/kube-rbac-proxy/0.log" Jul 14 19:09:51.866439 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.866407 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797996db74-bzk2c_8cbd1903-2698-4c5d-8e73-2910b6991f36/prom-label-proxy/0.log" Jul 14 19:09:51.889257 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.889226 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797996db74-bzk2c_8cbd1903-2698-4c5d-8e73-2910b6991f36/kube-rbac-proxy-rules/0.log" Jul 14 19:09:51.911341 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:51.911314 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-797996db74-bzk2c_8cbd1903-2698-4c5d-8e73-2910b6991f36/kube-rbac-proxy-metrics/0.log" Jul 14 19:09:52.770275 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.770242 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj"] Jul 14 19:09:52.780672 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.780644 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:52.783488 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.782715 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj"] Jul 14 19:09:52.897686 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.897653 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c155d252-fc3d-49e4-983a-ebaa3065624b-lib-modules\") pod \"perf-node-gather-daemonset-pbscj\" (UID: \"c155d252-fc3d-49e4-983a-ebaa3065624b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:52.897848 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.897831 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c155d252-fc3d-49e4-983a-ebaa3065624b-proc\") pod \"perf-node-gather-daemonset-pbscj\" (UID: \"c155d252-fc3d-49e4-983a-ebaa3065624b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:52.897923 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.897864 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c155d252-fc3d-49e4-983a-ebaa3065624b-podres\") pod \"perf-node-gather-daemonset-pbscj\" (UID: \"c155d252-fc3d-49e4-983a-ebaa3065624b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:52.897923 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.897907 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c155d252-fc3d-49e4-983a-ebaa3065624b-sys\") pod \"perf-node-gather-daemonset-pbscj\" (UID: \"c155d252-fc3d-49e4-983a-ebaa3065624b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:52.898051 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.897932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx2nd\" (UniqueName: \"kubernetes.io/projected/c155d252-fc3d-49e4-983a-ebaa3065624b-kube-api-access-hx2nd\") pod \"perf-node-gather-daemonset-pbscj\" (UID: \"c155d252-fc3d-49e4-983a-ebaa3065624b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:52.999419 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.999378 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c155d252-fc3d-49e4-983a-ebaa3065624b-sys\") pod \"perf-node-gather-daemonset-pbscj\" (UID: \"c155d252-fc3d-49e4-983a-ebaa3065624b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:52.999612 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.999434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2nd\" (UniqueName: \"kubernetes.io/projected/c155d252-fc3d-49e4-983a-ebaa3065624b-kube-api-access-hx2nd\") pod \"perf-node-gather-daemonset-pbscj\" (UID: \"c155d252-fc3d-49e4-983a-ebaa3065624b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:52.999612 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.999530 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c155d252-fc3d-49e4-983a-ebaa3065624b-lib-modules\") pod \"perf-node-gather-daemonset-pbscj\" (UID: \"c155d252-fc3d-49e4-983a-ebaa3065624b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:52.999731 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.999624 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c155d252-fc3d-49e4-983a-ebaa3065624b-proc\") pod \"perf-node-gather-daemonset-pbscj\" (UID: \"c155d252-fc3d-49e4-983a-ebaa3065624b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:52.999731 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.999651 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c155d252-fc3d-49e4-983a-ebaa3065624b-podres\") pod \"perf-node-gather-daemonset-pbscj\" (UID: \"c155d252-fc3d-49e4-983a-ebaa3065624b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:52.999860 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.999837 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c155d252-fc3d-49e4-983a-ebaa3065624b-podres\") pod \"perf-node-gather-daemonset-pbscj\" (UID: \"c155d252-fc3d-49e4-983a-ebaa3065624b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:52.999923 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.999903 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c155d252-fc3d-49e4-983a-ebaa3065624b-sys\") pod \"perf-node-gather-daemonset-pbscj\" (UID: \"c155d252-fc3d-49e4-983a-ebaa3065624b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:52.999966 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:52.999956 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c155d252-fc3d-49e4-983a-ebaa3065624b-proc\") pod \"perf-node-gather-daemonset-pbscj\" (UID: \"c155d252-fc3d-49e4-983a-ebaa3065624b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:53.000115 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:53.000094 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c155d252-fc3d-49e4-983a-ebaa3065624b-lib-modules\") pod \"perf-node-gather-daemonset-pbscj\" (UID: \"c155d252-fc3d-49e4-983a-ebaa3065624b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:53.009829 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:53.009801 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx2nd\" (UniqueName: \"kubernetes.io/projected/c155d252-fc3d-49e4-983a-ebaa3065624b-kube-api-access-hx2nd\") pod \"perf-node-gather-daemonset-pbscj\" (UID: \"c155d252-fc3d-49e4-983a-ebaa3065624b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:53.095882 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:53.095785 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:53.270322 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:53.270278 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj"] Jul 14 19:09:53.600864 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:53.600774 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/1.log" Jul 14 19:09:53.610045 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:53.610017 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-mwmlt_159e6d97-7ed3-42d8-b502-9add9b40eccb/console-operator/2.log" Jul 14 19:09:53.981956 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:53.981915 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" event={"ID":"c155d252-fc3d-49e4-983a-ebaa3065624b","Type":"ContainerStarted","Data":"fb39da614d0caab1e60787eb3d6c22f25b89888fea81d49c9d30c265f0f2e235"} Jul 14 19:09:53.982485 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:53.981983 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" event={"ID":"c155d252-fc3d-49e4-983a-ebaa3065624b","Type":"ContainerStarted","Data":"103b86d36b143e9b037b05987d3860597af98cddcffeb8786a7d28bae01431ce"} Jul 14 19:09:53.982485 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:53.982024 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:09:54.001470 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:54.001407 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" podStartSLOduration=2.001386596 podStartE2EDuration="2.001386596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:09:53.999504006 +0000 UTC m=+2895.536951792" watchObservedRunningTime="2026-07-14 19:09:54.001386596 +0000 UTC m=+2895.538834382" Jul 14 19:09:54.149542 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:54.149513 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-4t7qb_793e27fb-3087-4865-8e9d-f0461c048e1a/download-server/0.log" Jul 14 19:09:54.779400 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:54.779365 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-phr5m_d1f7e1f6-6948-4d4d-be15-eca40f74051f/volume-data-source-validator/0.log" Jul 14 19:09:55.876907 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:55.876877 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7v79r_3f139ad7-4fe0-4765-9a55-18231d6dd7cd/dns/0.log" Jul 14 19:09:55.899955 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:55.899924 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7v79r_3f139ad7-4fe0-4765-9a55-18231d6dd7cd/kube-rbac-proxy/0.log" Jul 14 19:09:56.076756 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:56.076727 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tjpz8_5bdb299c-c772-4af9-8286-6fb4c2371885/dns-node-resolver/0.log" Jul 14 19:09:56.713589 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:56.713553 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kjlrj_80ba9b7f-15c6-4fc0-82ab-8286c596aff1/node-ca/0.log" Jul 14 19:09:57.765498 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:57.765469 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4tgcnj_99b9b6de-ac61-470e-948e-e48e88cf4039/istio-proxy/0.log" Jul 14 19:09:58.062377 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:58.062279 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d44f677bf-zv7jq_b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c/kube-auth-proxy/0.log" Jul 14 19:09:58.094602 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:58.094569 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rlbm8_83981a26-6460-417f-9442-602373954802/istio-proxy/0.log" Jul 14 19:09:58.157797 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:58.157755 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rlbm8_83981a26-6460-417f-9442-602373954802/istio-proxy/1.log" Jul 14 19:09:58.238997 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:58.238964 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-k8tcp_64f2c200-c428-4c62-ad4c-d8f13482a641/payload-processing/0.log" Jul 14 19:09:58.268102 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:58.268069 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59f484b746-8hs6t_c7cf164f-807a-4191-acdd-e7d5f3c6de75/router/0.log" Jul 14 19:09:59.037953 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:59.037912 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjjpz_8e2729e0-7391-4488-8c08-285392a648be/serve-healthcheck-canary/0.log" Jul 14 19:09:59.687993 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:59.687963 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-tkrhd_6392abcd-2ccd-410b-8c10-08a58197237e/insights-operator/0.log" Jul 14 19:09:59.689333 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:59.689301 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-tkrhd_6392abcd-2ccd-410b-8c10-08a58197237e/insights-operator/1.log" Jul 14 19:09:59.902475 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:59.902447 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-znj4p_1091f487-d0e7-4d2d-96ac-1849adce0fa6/kube-rbac-proxy/0.log" Jul 14 19:09:59.929695 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:59.929670 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-znj4p_1091f487-d0e7-4d2d-96ac-1849adce0fa6/exporter/0.log" Jul 14 19:09:59.960720 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:09:59.960696 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-znj4p_1091f487-d0e7-4d2d-96ac-1849adce0fa6/extractor/0.log" Jul 14 19:10:00.008210 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:00.007505 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-pbscj" Jul 14 19:10:02.480647 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:02.480620 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-46m4j_8bdf5bf9-9f6b-40b7-8afd-1c6dd47dab0d/manager/0.log" Jul 14 19:10:02.747461 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:02.747337 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-knkxx_fcce408f-a62a-426c-afce-01250b0490ec/manager/0.log" Jul 14 19:10:02.918893 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:02.918853 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4tgcnj_99b9b6de-ac61-470e-948e-e48e88cf4039/istio-proxy/0.log" Jul 14 19:10:02.934990 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:02.934960 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-846475b4dc-cj4kt_ddf090f3-ed96-418b-85ef-9f6f25f387e6/manager/0.log" Jul 14 19:10:02.958392 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:02.958362 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-hxhf5_b9077472-e6b7-4938-a5c1-6acf578d0e13/server/0.log" Jul 14 19:10:03.085277 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:03.085240 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-fgbsf_460ba81a-42b2-48e0-9fb0-ac035c67fd36/manager/0.log" Jul 14 19:10:03.155900 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:03.155870 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d44f677bf-zv7jq_b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c/kube-auth-proxy/0.log" Jul 14 19:10:03.178236 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:03.178210 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rlbm8_83981a26-6460-417f-9442-602373954802/istio-proxy/0.log" Jul 14 19:10:03.272678 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:03.272640 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rlbm8_83981a26-6460-417f-9442-602373954802/istio-proxy/1.log" Jul 14 19:10:03.290612 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:03.290581 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d948cf9bc-5vfql_45be84e1-0678-4cb0-aaf1-8898001e3eeb/manager/0.log" Jul 14 19:10:03.309366 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:03.309268 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d948cf9bc-5vfql_45be84e1-0678-4cb0-aaf1-8898001e3eeb/copy-manifests/0.log" Jul 14 19:10:03.335419 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:03.335396 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-k8tcp_64f2c200-c428-4c62-ad4c-d8f13482a641/payload-processing/0.log" Jul 14 19:10:03.348739 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:03.348716 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59f484b746-8hs6t_c7cf164f-807a-4191-acdd-e7d5f3c6de75/router/0.log" Jul 14 19:10:04.592554 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:04.592470 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4tgcnj_99b9b6de-ac61-470e-948e-e48e88cf4039/istio-proxy/0.log" Jul 14 19:10:04.820432 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:04.820402 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d44f677bf-zv7jq_b46b6c3a-b3d0-47c4-b198-3c9c1e8fcb1c/kube-auth-proxy/0.log" Jul 14 19:10:04.848265 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:04.848178 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rlbm8_83981a26-6460-417f-9442-602373954802/istio-proxy/0.log" Jul 14 19:10:04.906160 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:04.906124 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rlbm8_83981a26-6460-417f-9442-602373954802/istio-proxy/1.log" Jul 14 19:10:04.975326 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:04.975297 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-k8tcp_64f2c200-c428-4c62-ad4c-d8f13482a641/payload-processing/0.log" Jul 14 19:10:04.991750 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:04.991714 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59f484b746-8hs6t_c7cf164f-807a-4191-acdd-e7d5f3c6de75/router/0.log" Jul 14 19:10:10.242845 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:10.242750 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-n22g8_42555e5b-8ae2-4677-a74a-bbacb3393030/kube-storage-version-migrator-operator/1.log" Jul 14 19:10:10.244559 ip-10-0-132-224 kubenswrapper[2573]: I0714 19:10:10.244532 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-n22g8_42555e5b-8ae2-4677-a74a-bbacb3393030/kube-storage-version-migrator-operator/0.log"