Jul 22 16:40:34.934384 ip-10-0-132-94 systemd[1]: Starting Kubernetes Kubelet... Jul 22 16:40:35.332957 ip-10-0-132-94 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 16:40:35.332957 ip-10-0-132-94 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 16:40:35.332957 ip-10-0-132-94 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 16:40:35.332957 ip-10-0-132-94 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 16:40:35.332957 ip-10-0-132-94 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 16:40:35.332957 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.216489 2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 16:40:35.332957 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316314 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:40:35.332957 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316329 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:40:35.332957 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316333 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:40:35.332957 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316336 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:40:35.332957 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316339 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:40:35.332957 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316342 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:40:35.332957 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316345 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:40:35.332957 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316348 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316351 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316353 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316356 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316359 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316361 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316373 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316379 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316383 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316387 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316390 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316393 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316396 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316399 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316401 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316404 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316407 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316410 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316413 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316416 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:40:35.334858 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316419 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316422 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316424 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316428 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316431 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316435 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316437 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316440 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316443 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316445 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316449 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316451 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316454 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316456 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316459 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316462 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316465 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316467 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316470 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316472 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:40:35.336996 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316475 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316477 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316480 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316482 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316485 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316487 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316490 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316492 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316495 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316497 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316499 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316502 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316504 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316507 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316509 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316514 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316516 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316519 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316521 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:40:35.343968 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316524 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316526 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316529 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316531 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316534 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316537 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316541 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316544 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316546 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316549 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316551 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316553 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316556 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316558 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316561 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316565 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316568 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316571 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316574 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316578 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:40:35.344881 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316580 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316583 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316586 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316588 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316591 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316594 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316596 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316599 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316603 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316605 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316608 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316610 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.316613 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317321 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317329 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317332 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317335 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317338 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317341 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:40:35.345689 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317344 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317347 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317350 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317353 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317355 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317358 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317361 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317363 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317366 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317368 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317371 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317375 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317379 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317382 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317385 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317388 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317390 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317393 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317396 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:40:35.347638 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317399 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317401 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317404 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317407 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317409 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317412 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317415 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317418 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317421 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317423 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317427 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317429 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317432 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317435 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317438 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317441 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317444 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317446 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317449 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317451 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:40:35.348508 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317454 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317456 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317459 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317461 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317464 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317467 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317469 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317472 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317474 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317477 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317479 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317482 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317484 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317487 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317490 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317493 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317498 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317501 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317504 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317506 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317509 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:40:35.350997 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317511 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317514 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317516 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317519 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317522 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317529 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317532 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317536 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317539 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317542 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317544 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317547 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317550 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317553 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317555 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317558 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317561 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317563 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317566 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:40:35.351895 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317568 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317571 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317573 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317576 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317578 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317581 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317584 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317586 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317591 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317594 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317596 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317599 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317602 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.317604 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318079 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318085 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318088 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318091 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318094 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318097 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:40:35.540524 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318100 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:40:35.385298 ip-10-0-132-94 systemd[1]: Started Kubernetes Kubelet. Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318103 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318106 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318109 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318112 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318114 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318117 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318119 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318122 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318125 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318127 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318130 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318132 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318137 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318139 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318142 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318144 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318147 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318150 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318152 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:40:35.556958 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318156 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318159 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318162 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318164 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318167 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318169 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318172 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318174 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318177 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318179 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318182 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318185 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318188 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318191 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318193 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318196 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318198 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318201 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318203 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318206 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:40:35.562805 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318209 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318211 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318214 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318216 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318219 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318223 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318225 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318228 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318231 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318233 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318236 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318238 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318242 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318245 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318247 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318250 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318253 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318255 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318258 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318261 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:40:35.564111 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318264 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318266 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318269 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318272 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318276 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318279 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318282 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318284 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318287 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318289 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318292 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318294 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318297 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318300 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318302 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318305 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318307 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318313 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318317 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:40:35.565136 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318320 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318322 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318325 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318329 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318332 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318337 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318340 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318343 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318346 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318348 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318351 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318354 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318357 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.318360 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318436 2557 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318444 2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318450 2557 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318455 2557 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318459 2557 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318463 2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 16:40:35.565952 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318468 2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318472 2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318475 2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318478 2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318482 2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318485 2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318488 2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318491 2557 flags.go:64] FLAG: --cgroup-root="" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318494 2557 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318497 2557 flags.go:64] FLAG: --client-ca-file="" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318501 2557 flags.go:64] FLAG: --cloud-provider="external" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318504 2557 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318508 2557 flags.go:64] FLAG: --cluster-domain="" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318511 2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318514 2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318517 2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318521 2557 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318526 2557 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318529 2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318532 2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318535 2557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318538 2557 flags.go:64] FLAG: --contention-profiling="false" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318541 2557 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318544 2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 16:40:35.566645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318547 2557 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318550 2557 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318554 2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318557 2557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318560 2557 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318563 2557 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318566 2557 flags.go:64] FLAG: --enable-server="true" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318570 2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318575 2557 flags.go:64] FLAG: --event-burst="100" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318578 2557 flags.go:64] FLAG: --event-qps="50" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318581 2557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318584 2557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318587 2557 flags.go:64] FLAG: --eviction-hard="" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318591 2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318593 2557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318597 2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318599 2557 flags.go:64] FLAG: --eviction-soft="" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318602 2557 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318607 2557 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318610 2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318613 2557 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318616 2557 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318619 2557 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318622 2557 flags.go:64] FLAG: --feature-gates="" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318625 2557 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 16:40:35.567521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318630 2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318633 2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318636 2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318639 2557 flags.go:64] FLAG: --healthz-port="10248" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318642 2557 flags.go:64] FLAG: --help="false" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318645 2557 flags.go:64] FLAG: --hostname-override="ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318648 2557 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318651 2557 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318654 2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318658 2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318661 2557 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318664 2557 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318667 2557 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318669 2557 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318672 2557 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318676 2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318679 2557 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318682 2557 flags.go:64] FLAG: --kube-reserved="" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318685 2557 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318688 2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318691 2557 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318693 2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318696 2557 flags.go:64] FLAG: --lock-file="" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318699 2557 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 16:40:35.568430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318702 2557 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318706 2557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318712 2557 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318715 2557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318717 2557 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318720 2557 flags.go:64] FLAG: --logging-format="text" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318723 2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318726 2557 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318730 2557 flags.go:64] FLAG: --manifest-url="" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318733 2557 flags.go:64] FLAG: --manifest-url-header="" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318738 2557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318741 2557 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318745 2557 flags.go:64] FLAG: --max-pods="110" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318748 2557 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318751 2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318753 2557 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318756 2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318759 2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318762 2557 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318765 2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318773 2557 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318776 2557 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318779 2557 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 16:40:35.569330 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318782 2557 flags.go:64] FLAG: --pod-cidr="" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318785 2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318791 2557 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318794 2557 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318797 2557 flags.go:64] FLAG: --pods-per-core="0" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318800 2557 flags.go:64] FLAG: --port="10250" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318803 2557 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318806 2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03965ec78102e362e" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318809 2557 flags.go:64] FLAG: --qos-reserved="" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318812 2557 flags.go:64] FLAG: --read-only-port="10255" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318816 2557 flags.go:64] FLAG: --register-node="true" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318820 2557 flags.go:64] FLAG: --register-with-taints="" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318823 2557 flags.go:64] FLAG: --registry-burst="10" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318826 2557 flags.go:64] FLAG: --registry-qps="5" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318829 2557 flags.go:64] FLAG: --reserved-cpus="" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318832 2557 flags.go:64] FLAG: --reserved-memory="" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318835 2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318840 2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318843 2557 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318846 2557 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318849 2557 flags.go:64] FLAG: --runonce="false" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318852 2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318855 2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318858 2557 flags.go:64] FLAG: --seccomp-default="false" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318860 2557 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 16:40:35.570306 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318863 2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318866 2557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318869 2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318872 2557 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318875 2557 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318878 2557 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318881 2557 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318884 2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318887 2557 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318891 2557 flags.go:64] FLAG: --system-cgroups="" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318893 2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318899 2557 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318901 2557 flags.go:64] FLAG: --tls-cert-file="" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318904 2557 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318908 2557 flags.go:64] FLAG: --tls-min-version="" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318911 2557 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318913 2557 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318916 2557 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318920 2557 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318923 2557 flags.go:64] FLAG: --v="2" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318927 2557 flags.go:64] FLAG: --version="false" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318931 2557 flags.go:64] FLAG: --vmodule="" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318934 2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.318937 2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319076 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:40:35.571105 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319080 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319083 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319086 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319089 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319092 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319095 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319097 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319100 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319102 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319106 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319110 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319113 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319116 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319119 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319122 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319125 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319128 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319131 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319134 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:40:35.571822 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319136 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319139 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319141 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319144 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319147 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319149 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319153 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319156 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319158 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319161 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319164 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319166 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319170 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319172 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319175 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319179 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319182 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319185 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319189 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:40:35.572393 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319191 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319194 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319197 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319200 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319202 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319205 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319208 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319210 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319213 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319216 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319218 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319221 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319224 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319227 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319229 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319232 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319234 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319237 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319239 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319243 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:40:35.572867 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319246 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319249 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319251 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319254 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319256 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319261 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319263 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319266 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319268 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319271 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319274 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319277 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319279 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319282 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319284 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319287 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319289 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319292 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319294 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:40:35.573377 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319297 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319300 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319302 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319305 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319308 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319310 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319313 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319315 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319318 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319322 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319325 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319328 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319332 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319334 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319337 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319339 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319342 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319344 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319348 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319351 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:40:35.573853 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.319354 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:40:35.574440 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.319925 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 16:40:35.574440 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.326078 2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 16:40:35.574440 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.326094 2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 16:40:35.574440 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.326115 2557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 16:40:35.574440 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.326124 2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 16:40:35.574440 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326180 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:40:35.574440 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326185 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:40:35.574440 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326188 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:40:35.574440 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326191 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:40:35.574440 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326195 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:40:35.574440 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326197 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:40:35.574440 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326200 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:40:35.574440 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326203 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:40:35.574440 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326205 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:40:35.574440 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326208 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326210 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326214 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326218 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326221 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326223 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326226 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326229 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326232 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326234 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326237 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326239 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326242 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326244 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326247 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326250 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326252 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326255 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326257 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326260 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:40:35.574876 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326263 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326266 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326268 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326271 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326273 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326276 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326279 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326281 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326284 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326288 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326292 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326295 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326297 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326300 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326302 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326304 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326308 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326311 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326313 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:40:35.575620 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326319 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326322 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326324 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326328 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326332 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326335 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326338 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326341 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326344 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326346 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326349 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326351 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326354 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326356 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326360 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326362 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326365 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326367 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326370 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326372 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:40:35.576335 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326375 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326377 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326380 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326382 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326385 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326387 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326389 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326392 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326394 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326397 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326401 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326405 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326409 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326412 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326415 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326418 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326420 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326423 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326425 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:40:35.577075 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326428 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:40:35.577783 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326431 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:40:35.577783 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326433 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:40:35.577783 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326436 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:40:35.577783 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326439 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:40:35.577783 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326442 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:40:35.577783 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326445 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:40:35.577783 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326447 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:40:35.577783 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326450 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:40:35.577783 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326452 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:40:35.577783 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326455 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:40:35.577783 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326457 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:40:35.577783 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.326462 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 16:40:35.577783 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326574 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 16:40:35.577783 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326578 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 16:40:35.577783 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326581 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326584 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326587 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326590 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326593 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326596 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326599 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326602 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326605 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326608 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326612 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326615 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326617 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326620 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326622 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326625 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326627 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326631 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326636 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 16:40:35.578344 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326639 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326642 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326645 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326647 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326649 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326652 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326654 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326657 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326659 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326662 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326664 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326667 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326669 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326672 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326675 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326677 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326680 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326683 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326686 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 16:40:35.579072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326689 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326691 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326694 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326696 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326701 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326703 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326705 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326708 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326711 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326713 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326716 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326718 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326723 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326726 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326729 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326731 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326733 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326736 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326738 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326741 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 16:40:35.579866 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326743 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326746 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326749 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326753 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326756 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326758 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326761 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326764 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326766 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326769 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326771 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326774 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326776 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326779 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326781 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326784 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326788 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326791 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 16:40:35.580710 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326793 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326796 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326798 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326800 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326803 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326805 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326809 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326812 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326814 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326817 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326819 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326822 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326825 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326827 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326829 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326832 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326834 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326838 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326840 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326842 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 16:40:35.581398 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:35.326845 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 16:40:35.582147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.326850 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 16:40:35.582147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.326952 2557 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 16:40:35.582147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.329238 2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 16:40:35.582147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.331615 2557 server.go:1019] "Starting client certificate rotation" Jul 22 16:40:35.582147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.331981 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 16:40:35.582147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.332018 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 16:40:35.582147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.336360 2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 16:40:35.582147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.342415 2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 16:40:35.582147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.353520 2557 log.go:25] "Validated CRI v1 runtime API" Jul 22 16:40:35.582147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.355318 2557 log.go:25] "Validated CRI v1 image API" Jul 22 16:40:35.582147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.356116 2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 16:40:35.582147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.362376 2557 fs.go:135] Filesystem UUIDs: map[096e28e8-1b65-4d8f-9390-dd9d750a2e03:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fb7b9f3e-7496-4841-a892-dd0b5fbcb181:/dev/nvme0n1p3] Jul 22 16:40:35.582598 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.362391 2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 16:40:35.582598 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.364085 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.367931 2557 manager.go:216] Machine: {Timestamp:2026-07-22 16:40:35.366894268 +0000 UTC m=+0.343622574 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099259 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cc9e83b88106f270b4a319f92acaf SystemUUID:ec2cc9e8-3b88-106f-270b-4a319f92acaf BootID:82ae8832-4836-4452-8bac-9743cfaadb1d Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d7:ed:67:c5:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d7:ed:67:c5:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:9c:fe:d4:cc:4e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.368048 2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.368157 2557 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.370181 2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.370206 2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-94.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.371072 2557 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.371082 2557 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.371090 2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.371522 2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.372435 2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.374628 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.374736 2557 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.374763 2557 kubelet.go:486] "Attempting to sync node with API server" Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.374772 2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.374782 2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.374791 2557 kubelet.go:398] "Adding apiserver pod source" Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.374809 2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 16:40:35.582690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.375126 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.375141 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.377129 2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.379214 2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.379242 2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.380101 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.380115 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.380121 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.380126 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.380135 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.380140 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.380148 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.380154 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.380160 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.380166 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.380180 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.380186 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.380670 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.380676 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.384258 2557 server.go:1268] "Started kubelet" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.384299 2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.384362 2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.384422 2557 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.386803 2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-94.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.386871 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.386877 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-94.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.388567 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-clldw" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.388817 2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.389537 2557 server.go:310] "Adding debug handlers to kubelet server" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.390870 2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.390640 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-94.ec2.internal.18c4aa26aa0ed7aa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-94.ec2.internal,UID:ip-10-0-132-94.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-94.ec2.internal,},FirstTimestamp:2026-07-22 16:40:35.384137642 +0000 UTC m=+0.360865949,LastTimestamp:2026-07-22 16:40:35.384137642 +0000 UTC m=+0.360865949,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-94.ec2.internal,}" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.395482 2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.395536 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.395560 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-94.ec2.internal\" not found" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.396017 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-clldw" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.396212 2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.396219 2557 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.396248 2557 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.396316 2557 reconstruct.go:97] "Volume reconstruction finished" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.396323 2557 reconciler.go:29] "Reconciler: start to sync state" Jul 22 16:40:35.583421 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.396327 2557 factory.go:55] Registering systemd factory Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.396340 2557 factory.go:223] Registration of the systemd container factory successfully Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.396547 2557 factory.go:153] Registering CRI-O factory Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.396558 2557 factory.go:223] Registration of the crio container factory successfully Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.396597 2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.396615 2557 factory.go:103] Registering Raw factory Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.396626 2557 manager.go:1196] Started watching for new ooms in manager Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.396998 2557 manager.go:349] Starting recovery of all containers Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.402627 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.403800 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.404216 2557 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-94.ec2.internal\" not found" node="ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.407257 2557 manager.go:354] Recovery completed Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.412152 2557 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.415122 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.416846 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.416866 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.416876 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.417462 2557 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.417469 2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.417484 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.418323 2557 policy_none.go:49] "None policy: Start" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.418335 2557 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.418345 2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.418971 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.419147 2557 policy_none.go:47] "Start" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.419698 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.419726 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.419748 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.464595 2557 manager.go:337] "Starting Device Plugin manager" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.464624 2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.464633 2557 server.go:87] "Starting device plugin registration server" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.464879 2557 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.464892 2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.465259 2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.465490 2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.465500 2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.465731 2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.465768 2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-94.ec2.internal\" not found" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.565458 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.566315 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.566345 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.566361 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.566397 2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.568447 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.568485 2557 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.568506 2557 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.568543 2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.570505 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:40:35.584953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.574773 2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.586755 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.574796 2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-94.ec2.internal\": node \"ip-10-0-132-94.ec2.internal\" not found" Jul 22 16:40:35.595729 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.595707 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-94.ec2.internal\" not found" Jul 22 16:40:35.669041 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.668980 2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-94.ec2.internal"] Jul 22 16:40:35.669169 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.669101 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:40:35.670775 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.670755 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:40:35.670877 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.670789 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:40:35.670877 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.670803 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:40:35.673219 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.673202 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:40:35.673345 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.673328 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.673412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.673367 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:40:35.674135 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.674119 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:40:35.674200 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.674143 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:40:35.674200 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.674156 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:40:35.674397 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.674382 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:40:35.674397 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.674401 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:40:35.674552 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.674413 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:40:35.676679 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.676666 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.676741 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.676688 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 16:40:35.679899 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.679877 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasSufficientMemory" Jul 22 16:40:35.679973 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.679912 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 16:40:35.679973 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.679930 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeHasSufficientPID" Jul 22 16:40:35.696645 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.696625 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-94.ec2.internal\" not found" Jul 22 16:40:35.697611 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.697597 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-94.ec2.internal\" not found" node="ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.697955 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.697938 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/466327f919d9276f8f0590ca18d52c6c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-94.ec2.internal\" (UID: \"466327f919d9276f8f0590ca18d52c6c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.698043 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.697967 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/174c6948cb8527f74edd0b4217d9cd0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal\" (UID: \"174c6948cb8527f74edd0b4217d9cd0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.698043 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.697992 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/174c6948cb8527f74edd0b4217d9cd0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal\" (UID: \"174c6948cb8527f74edd0b4217d9cd0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.704857 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.704839 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-94.ec2.internal\" not found" node="ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.797783 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.797751 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-94.ec2.internal\" not found" Jul 22 16:40:35.798974 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.798955 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/174c6948cb8527f74edd0b4217d9cd0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal\" (UID: \"174c6948cb8527f74edd0b4217d9cd0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.799049 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.798983 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/466327f919d9276f8f0590ca18d52c6c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-94.ec2.internal\" (UID: \"466327f919d9276f8f0590ca18d52c6c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.799049 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.799008 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/174c6948cb8527f74edd0b4217d9cd0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal\" (UID: \"174c6948cb8527f74edd0b4217d9cd0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.799171 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.799062 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/174c6948cb8527f74edd0b4217d9cd0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal\" (UID: \"174c6948cb8527f74edd0b4217d9cd0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.799171 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.799092 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/466327f919d9276f8f0590ca18d52c6c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-94.ec2.internal\" (UID: \"466327f919d9276f8f0590ca18d52c6c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.799171 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:35.799098 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/174c6948cb8527f74edd0b4217d9cd0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal\" (UID: \"174c6948cb8527f74edd0b4217d9cd0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal" Jul 22 16:40:35.898907 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.898818 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-94.ec2.internal\" not found" Jul 22 16:40:35.999716 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:35.999682 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-94.ec2.internal\" not found" Jul 22 16:40:36.000904 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:36.000890 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal" Jul 22 16:40:36.007144 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:36.007123 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-94.ec2.internal" Jul 22 16:40:36.099791 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:36.099752 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-94.ec2.internal\" not found" Jul 22 16:40:36.200694 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:36.200626 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-94.ec2.internal\" not found" Jul 22 16:40:36.301312 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:36.301280 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-94.ec2.internal\" not found" Jul 22 16:40:36.332929 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:36.332897 2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 16:40:36.333109 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:36.333085 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 16:40:36.333486 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:36.333125 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 16:40:36.399106 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:36.399047 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 16:35:36 +0000 UTC" deadline="2028-01-14 12:52:24.696060152 +0000 UTC" Jul 22 16:40:36.399106 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:36.399095 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12980h11m48.29697029s" Jul 22 16:40:36.401970 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:36.401945 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-94.ec2.internal\" not found" Jul 22 16:40:36.502752 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:36.502720 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-94.ec2.internal\" not found" Jul 22 16:40:36.603915 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:36.603880 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-94.ec2.internal\" not found" Jul 22 16:40:36.623160 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:36.623119 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174c6948cb8527f74edd0b4217d9cd0d.slice/crio-0acaef543e0e2d431471472f72063d36a220a9232b9e8a9b57b904de77fee01d WatchSource:0}: Error finding container 0acaef543e0e2d431471472f72063d36a220a9232b9e8a9b57b904de77fee01d: Status 404 returned error can't find the container with id 0acaef543e0e2d431471472f72063d36a220a9232b9e8a9b57b904de77fee01d Jul 22 16:40:36.623469 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:36.623441 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466327f919d9276f8f0590ca18d52c6c.slice/crio-5f521245f7902e574621315d54c66637e8ef6de76124bd35e78e33dc040daae1 WatchSource:0}: Error finding container 5f521245f7902e574621315d54c66637e8ef6de76124bd35e78e33dc040daae1: Status 404 returned error can't find the container with id 5f521245f7902e574621315d54c66637e8ef6de76124bd35e78e33dc040daae1 Jul 22 16:40:36.628376 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:36.628358 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:40:36.654162 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:36.653566 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:40:36.704903 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:36.704874 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-94.ec2.internal\" not found" Jul 22 16:40:36.755068 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:36.754996 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:40:36.798547 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:36.798507 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal" Jul 22 16:40:36.809864 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:36.809839 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 16:40:36.810592 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:36.810579 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-94.ec2.internal" Jul 22 16:40:36.820578 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:36.820559 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 16:40:37.376156 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.376123 2557 apiserver.go:52] "Watching apiserver" Jul 22 16:40:37.384565 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.384540 2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 16:40:37.385065 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.385026 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-4842j","kube-system/kube-apiserver-proxy-ip-10-0-132-94.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7","openshift-cluster-node-tuning-operator/tuned-kngsc","openshift-dns/node-resolver-f7wc2","openshift-image-registry/node-ca-hfg9n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal","openshift-multus/multus-qb6fj","kube-system/konnectivity-agent-4lmvp","openshift-multus/multus-additional-cni-plugins-ncfqd","openshift-multus/network-metrics-daemon-65bkg","openshift-network-diagnostics/network-check-target-kqtz5","openshift-network-operator/iptables-alerter-wjjfr","openshift-ovn-kubernetes/ovnkube-node-2bs4s"] Jul 22 16:40:37.388014 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.387993 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.390794 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.390766 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wjjfr" Jul 22 16:40:37.391321 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.391286 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 16:40:37.391456 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.391344 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 16:40:37.391456 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.391380 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 16:40:37.391635 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.391292 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7xfxb\"" Jul 22 16:40:37.391635 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.391530 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 16:40:37.391797 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.391710 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 16:40:37.394691 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.394665 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.395489 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.395472 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rk5gw\"" Jul 22 16:40:37.395620 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.395565 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 16:40:37.395620 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.395577 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 16:40:37.395771 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.395670 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:40:37.395861 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.395835 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.396774 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.396749 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 16:40:37.397571 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.397547 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jz65p\"" Jul 22 16:40:37.397810 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.397793 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 16:40:37.397887 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.397840 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 16:40:37.398692 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.398113 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 16:40:37.398692 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.398314 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:40:37.398692 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.398357 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 16:40:37.398692 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.398685 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c545c\"" Jul 22 16:40:37.399779 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.399753 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f7wc2" Jul 22 16:40:37.402466 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.402424 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 16:40:37.402567 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.402452 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 16:40:37.402567 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.402551 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xh5fr\"" Jul 22 16:40:37.407485 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.407461 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hfg9n" Jul 22 16:40:37.408588 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.408341 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.408588 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.408373 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7163961-e9d7-4963-804c-f906fa9f8817-iptables-alerter-script\") pod \"iptables-alerter-wjjfr\" (UID: \"c7163961-e9d7-4963-804c-f906fa9f8817\") " pod="openshift-network-operator/iptables-alerter-wjjfr" Jul 22 16:40:37.408588 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.408403 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-sys-fs\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.408588 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.408428 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-modprobe-d\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.408588 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.408450 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-systemd\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.408588 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.408473 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-tmp\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.408588 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.408498 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-os-release\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.408588 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.408523 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.410309 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410281 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.410406 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410339 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7163961-e9d7-4963-804c-f906fa9f8817-host-slash\") pod \"iptables-alerter-wjjfr\" (UID: \"c7163961-e9d7-4963-804c-f906fa9f8817\") " pod="openshift-network-operator/iptables-alerter-wjjfr" Jul 22 16:40:37.410406 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410393 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-device-dir\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.410521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410428 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-sys\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.410521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410459 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.410521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410515 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrbl\" (UniqueName: \"kubernetes.io/projected/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-kube-api-access-dbrbl\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.410668 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410550 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-registration-dir\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.410668 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410582 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6acdc380-0278-49e9-a977-f16754ea4d48-hosts-file\") pod \"node-resolver-f7wc2\" (UID: \"6acdc380-0278-49e9-a977-f16754ea4d48\") " pod="openshift-dns/node-resolver-f7wc2" Jul 22 16:40:37.410668 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410616 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-cnibin\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.410668 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410645 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rfzl\" (UniqueName: \"kubernetes.io/projected/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-kube-api-access-4rfzl\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.410886 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410677 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-sysctl-conf\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.410886 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410720 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-tuned\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.410886 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410763 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv6h5\" (UniqueName: \"kubernetes.io/projected/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-kube-api-access-xv6h5\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.410886 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410808 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6acdc380-0278-49e9-a977-f16754ea4d48-tmp-dir\") pod \"node-resolver-f7wc2\" (UID: \"6acdc380-0278-49e9-a977-f16754ea4d48\") " pod="openshift-dns/node-resolver-f7wc2" Jul 22 16:40:37.410886 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410857 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-system-cni-dir\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.411147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410893 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-run\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.411147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410929 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.411147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.410975 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-etc-selinux\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.411147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.411014 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-kubernetes\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.411147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.411041 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-sysctl-d\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.411147 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.411079 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-lib-modules\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.411454 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.411169 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-var-lib-kubelet\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.411454 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.411217 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5cbr\" (UniqueName: \"kubernetes.io/projected/6acdc380-0278-49e9-a977-f16754ea4d48-kube-api-access-c5cbr\") pod \"node-resolver-f7wc2\" (UID: \"6acdc380-0278-49e9-a977-f16754ea4d48\") " pod="openshift-dns/node-resolver-f7wc2" Jul 22 16:40:37.411454 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.411250 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw97z\" (UniqueName: \"kubernetes.io/projected/c7163961-e9d7-4963-804c-f906fa9f8817-kube-api-access-zw97z\") pod \"iptables-alerter-wjjfr\" (UID: \"c7163961-e9d7-4963-804c-f906fa9f8817\") " pod="openshift-network-operator/iptables-alerter-wjjfr" Jul 22 16:40:37.411454 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.411279 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-socket-dir\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.411454 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.411312 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-sysconfig\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.411454 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.411354 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-host\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.412016 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.411992 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 16:40:37.412149 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.412085 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 16:40:37.412492 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.412358 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f6hjb\"" Jul 22 16:40:37.412492 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.411992 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 16:40:37.412617 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.412554 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 16:40:37.413755 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.413251 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4lmvp" Jul 22 16:40:37.416360 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.416330 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 16:40:37.416460 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.416428 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pkb2j\"" Jul 22 16:40:37.416548 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.416505 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 16:40:37.420304 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.420272 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.422852 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.422832 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.423302 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.423281 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8zlgc\"" Jul 22 16:40:37.423517 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.423500 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 16:40:37.423880 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.423831 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 16:40:37.423880 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.423839 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 16:40:37.423880 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.423843 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 16:40:37.424028 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.423870 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 16:40:37.424200 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.424181 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 16:40:37.425757 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.425321 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:37.425757 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:37.425422 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:40:37.425916 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.425814 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 16:40:37.425984 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.425914 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fx4tc\"" Jul 22 16:40:37.428458 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.427997 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:37.428458 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:37.428102 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:40:37.430790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.430684 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:37.430790 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:37.430748 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:40:37.436503 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.436482 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pnc6p" Jul 22 16:40:37.445049 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.445028 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pnc6p" Jul 22 16:40:37.497964 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.497940 2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 16:40:37.511629 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.511596 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.511755 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.511638 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f202c1ae-315f-4f19-b0a3-706a278e1177-cni-binary-copy\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.511755 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.511668 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-log-socket\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.511755 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.511695 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.511755 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.511724 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.511755 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.511749 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-device-dir\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.512007 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.511777 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-etc-kubernetes\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.512007 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.511803 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-run-ovn-kubernetes\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.512007 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.511828 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs\") pod \"network-metrics-daemon-65bkg\" (UID: \"9704c18f-5f2d-417b-b49f-bf27e2774eef\") " pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:37.512007 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.511845 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.512007 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.511852 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-var-lib-cni-bin\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.512007 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.511923 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-run-multus-certs\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.512007 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.511962 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-device-dir\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512002 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-run-netns\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512063 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-run-systemd\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512098 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6acdc380-0278-49e9-a977-f16754ea4d48-hosts-file\") pod \"node-resolver-f7wc2\" (UID: \"6acdc380-0278-49e9-a977-f16754ea4d48\") " pod="openshift-dns/node-resolver-f7wc2" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512127 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-cnibin\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512153 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0c4cd2b-33ca-47fe-8263-dab2a28569cc-host\") pod \"node-ca-hfg9n\" (UID: \"f0c4cd2b-33ca-47fe-8263-dab2a28569cc\") " pod="openshift-image-registry/node-ca-hfg9n" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512182 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-cnibin\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512196 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6acdc380-0278-49e9-a977-f16754ea4d48-hosts-file\") pod \"node-resolver-f7wc2\" (UID: \"6acdc380-0278-49e9-a977-f16754ea4d48\") " pod="openshift-dns/node-resolver-f7wc2" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512229 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-cnibin\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512261 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-var-lib-openvswitch\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512286 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-system-cni-dir\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512327 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-multus-socket-dir-parent\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512348 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-hostroot\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512345 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-system-cni-dir\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512366 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512403 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-kubernetes\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.512412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512414 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512436 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-sysctl-d\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512464 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-lib-modules\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512493 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-var-lib-kubelet\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512503 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-kubernetes\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512518 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-node-log\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512548 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-sysconfig\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512555 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512571 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-host\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512599 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-var-lib-cni-multus\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512641 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-slash\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512661 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-lib-modules\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512665 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-var-lib-kubelet\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512669 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7163961-e9d7-4963-804c-f906fa9f8817-iptables-alerter-script\") pod \"iptables-alerter-wjjfr\" (UID: \"c7163961-e9d7-4963-804c-f906fa9f8817\") " pod="openshift-network-operator/iptables-alerter-wjjfr" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512708 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-sysconfig\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512709 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-sys-fs\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512739 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-systemd\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.513310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512750 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-sys-fs\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512768 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-run-openvswitch\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512779 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-sysctl-d\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512797 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-systemd\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512797 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/362a0176-19e5-42d8-808c-e32e578f7527-ovnkube-config\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512831 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-host\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512845 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/362a0176-19e5-42d8-808c-e32e578f7527-ovn-node-metrics-cert\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512879 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7163961-e9d7-4963-804c-f906fa9f8817-host-slash\") pod \"iptables-alerter-wjjfr\" (UID: \"c7163961-e9d7-4963-804c-f906fa9f8817\") " pod="openshift-network-operator/iptables-alerter-wjjfr" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512905 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-sys\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512915 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7163961-e9d7-4963-804c-f906fa9f8817-host-slash\") pod \"iptables-alerter-wjjfr\" (UID: \"c7163961-e9d7-4963-804c-f906fa9f8817\") " pod="openshift-network-operator/iptables-alerter-wjjfr" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512930 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0c4cd2b-33ca-47fe-8263-dab2a28569cc-serviceca\") pod \"node-ca-hfg9n\" (UID: \"f0c4cd2b-33ca-47fe-8263-dab2a28569cc\") " pod="openshift-image-registry/node-ca-hfg9n" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512955 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-os-release\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512961 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-sys\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.512978 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-run-netns\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513002 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-cni-bin\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513040 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-cni-netd\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513076 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/362a0176-19e5-42d8-808c-e32e578f7527-ovnkube-script-lib\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.514071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513144 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513175 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrbl\" (UniqueName: \"kubernetes.io/projected/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-kube-api-access-dbrbl\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513200 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-registration-dir\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513208 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7163961-e9d7-4963-804c-f906fa9f8817-iptables-alerter-script\") pod \"iptables-alerter-wjjfr\" (UID: \"c7163961-e9d7-4963-804c-f906fa9f8817\") " pod="openshift-network-operator/iptables-alerter-wjjfr" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513241 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-889s9\" (UniqueName: \"kubernetes.io/projected/f0c4cd2b-33ca-47fe-8263-dab2a28569cc-kube-api-access-889s9\") pod \"node-ca-hfg9n\" (UID: \"f0c4cd2b-33ca-47fe-8263-dab2a28569cc\") " pod="openshift-image-registry/node-ca-hfg9n" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513268 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-run-k8s-cni-cncf-io\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513292 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l5pp\" (UniqueName: \"kubernetes.io/projected/f202c1ae-315f-4f19-b0a3-706a278e1177-kube-api-access-8l5pp\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513320 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c831c7ba-6c28-4355-a3e1-271edff8c1ad-konnectivity-ca\") pod \"konnectivity-agent-4lmvp\" (UID: \"c831c7ba-6c28-4355-a3e1-271edff8c1ad\") " pod="kube-system/konnectivity-agent-4lmvp" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513348 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-run-ovn\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513391 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rfzl\" (UniqueName: \"kubernetes.io/projected/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-kube-api-access-4rfzl\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513416 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-sysctl-conf\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513441 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-tuned\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513466 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6h5\" (UniqueName: \"kubernetes.io/projected/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-kube-api-access-xv6h5\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513490 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-multus-cni-dir\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513580 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-registration-dir\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513637 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c831c7ba-6c28-4355-a3e1-271edff8c1ad-agent-certs\") pod \"konnectivity-agent-4lmvp\" (UID: \"c831c7ba-6c28-4355-a3e1-271edff8c1ad\") " pod="kube-system/konnectivity-agent-4lmvp" Jul 22 16:40:37.514905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513649 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513665 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-etc-openvswitch\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513702 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27f20e9c-87e3-4473-ad01-99525c7b95e1-kubelet-config\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513737 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6acdc380-0278-49e9-a977-f16754ea4d48-tmp-dir\") pod \"node-resolver-f7wc2\" (UID: \"6acdc380-0278-49e9-a977-f16754ea4d48\") " pod="openshift-dns/node-resolver-f7wc2" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513762 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-run\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513787 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f202c1ae-315f-4f19-b0a3-706a278e1177-multus-daemon-config\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513811 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-kubelet\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513837 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn87m\" (UniqueName: \"kubernetes.io/projected/362a0176-19e5-42d8-808c-e32e578f7527-kube-api-access-bn87m\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513864 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513934 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-etc-selinux\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513962 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-multus-conf-dir\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513971 2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.513988 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/362a0176-19e5-42d8-808c-e32e578f7527-env-overrides\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.514014 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwdtr\" (UniqueName: \"kubernetes.io/projected/9704c18f-5f2d-417b-b49f-bf27e2774eef-kube-api-access-kwdtr\") pod \"network-metrics-daemon-65bkg\" (UID: \"9704c18f-5f2d-417b-b49f-bf27e2774eef\") " pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.514026 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-sysctl-conf\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.514055 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5cbr\" (UniqueName: \"kubernetes.io/projected/6acdc380-0278-49e9-a977-f16754ea4d48-kube-api-access-c5cbr\") pod \"node-resolver-f7wc2\" (UID: \"6acdc380-0278-49e9-a977-f16754ea4d48\") " pod="openshift-dns/node-resolver-f7wc2" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.514086 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw97z\" (UniqueName: \"kubernetes.io/projected/c7163961-e9d7-4963-804c-f906fa9f8817-kube-api-access-zw97z\") pod \"iptables-alerter-wjjfr\" (UID: \"c7163961-e9d7-4963-804c-f906fa9f8817\") " pod="openshift-network-operator/iptables-alerter-wjjfr" Jul 22 16:40:37.515729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.514139 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-socket-dir\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.514166 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkb4\" (UniqueName: \"kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4\") pod \"network-check-target-kqtz5\" (UID: \"481c8a32-3270-4720-97fe-49de4a613919\") " pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.514190 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27f20e9c-87e3-4473-ad01-99525c7b95e1-dbus\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.514216 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.514242 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-modprobe-d\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.514267 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-tmp\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.514364 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6acdc380-0278-49e9-a977-f16754ea4d48-tmp-dir\") pod \"node-resolver-f7wc2\" (UID: \"6acdc380-0278-49e9-a977-f16754ea4d48\") " pod="openshift-dns/node-resolver-f7wc2" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.514790 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-system-cni-dir\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.514834 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-var-lib-kubelet\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.514971 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-modprobe-d\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.515064 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-etc-selinux\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.515170 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-systemd-units\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.515185 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-socket-dir\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.515219 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-os-release\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.515387 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-run\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.515397 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.516656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.515388 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-os-release\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.518185 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.518140 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-tmp\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.518308 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.518212 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-etc-tuned\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.524241 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.524217 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrbl\" (UniqueName: \"kubernetes.io/projected/ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6-kube-api-access-dbrbl\") pod \"multus-additional-cni-plugins-ncfqd\" (UID: \"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6\") " pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.524677 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.524654 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5cbr\" (UniqueName: \"kubernetes.io/projected/6acdc380-0278-49e9-a977-f16754ea4d48-kube-api-access-c5cbr\") pod \"node-resolver-f7wc2\" (UID: \"6acdc380-0278-49e9-a977-f16754ea4d48\") " pod="openshift-dns/node-resolver-f7wc2" Jul 22 16:40:37.525043 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.524997 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rfzl\" (UniqueName: \"kubernetes.io/projected/7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0-kube-api-access-4rfzl\") pod \"aws-ebs-csi-driver-node-pcgp7\" (UID: \"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.525043 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.525039 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw97z\" (UniqueName: \"kubernetes.io/projected/c7163961-e9d7-4963-804c-f906fa9f8817-kube-api-access-zw97z\") pod \"iptables-alerter-wjjfr\" (UID: \"c7163961-e9d7-4963-804c-f906fa9f8817\") " pod="openshift-network-operator/iptables-alerter-wjjfr" Jul 22 16:40:37.526471 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.526448 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6h5\" (UniqueName: \"kubernetes.io/projected/52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97-kube-api-access-xv6h5\") pod \"tuned-kngsc\" (UID: \"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97\") " pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.574551 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.574497 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-94.ec2.internal" event={"ID":"466327f919d9276f8f0590ca18d52c6c","Type":"ContainerStarted","Data":"5f521245f7902e574621315d54c66637e8ef6de76124bd35e78e33dc040daae1"} Jul 22 16:40:37.576068 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.576024 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal" event={"ID":"174c6948cb8527f74edd0b4217d9cd0d","Type":"ContainerStarted","Data":"0acaef543e0e2d431471472f72063d36a220a9232b9e8a9b57b904de77fee01d"} Jul 22 16:40:37.616280 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616254 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-run-openvswitch\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.616424 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616293 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/362a0176-19e5-42d8-808c-e32e578f7527-ovnkube-config\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.616424 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616320 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/362a0176-19e5-42d8-808c-e32e578f7527-ovn-node-metrics-cert\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.616424 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616369 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0c4cd2b-33ca-47fe-8263-dab2a28569cc-serviceca\") pod \"node-ca-hfg9n\" (UID: \"f0c4cd2b-33ca-47fe-8263-dab2a28569cc\") " pod="openshift-image-registry/node-ca-hfg9n" Jul 22 16:40:37.616424 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616394 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-os-release\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.616424 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616418 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-run-netns\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.616696 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616436 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-cni-bin\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.616696 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616452 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-cni-netd\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.616696 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616467 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/362a0176-19e5-42d8-808c-e32e578f7527-ovnkube-script-lib\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.616696 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616485 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-889s9\" (UniqueName: \"kubernetes.io/projected/f0c4cd2b-33ca-47fe-8263-dab2a28569cc-kube-api-access-889s9\") pod \"node-ca-hfg9n\" (UID: \"f0c4cd2b-33ca-47fe-8263-dab2a28569cc\") " pod="openshift-image-registry/node-ca-hfg9n" Jul 22 16:40:37.616696 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616507 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-run-k8s-cni-cncf-io\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.616696 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616531 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l5pp\" (UniqueName: \"kubernetes.io/projected/f202c1ae-315f-4f19-b0a3-706a278e1177-kube-api-access-8l5pp\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.616696 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616556 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c831c7ba-6c28-4355-a3e1-271edff8c1ad-konnectivity-ca\") pod \"konnectivity-agent-4lmvp\" (UID: \"c831c7ba-6c28-4355-a3e1-271edff8c1ad\") " pod="kube-system/konnectivity-agent-4lmvp" Jul 22 16:40:37.616696 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616581 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-run-ovn\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.616696 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616609 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-multus-cni-dir\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.616696 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616633 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c831c7ba-6c28-4355-a3e1-271edff8c1ad-agent-certs\") pod \"konnectivity-agent-4lmvp\" (UID: \"c831c7ba-6c28-4355-a3e1-271edff8c1ad\") " pod="kube-system/konnectivity-agent-4lmvp" Jul 22 16:40:37.616696 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616656 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-etc-openvswitch\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.616696 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616681 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27f20e9c-87e3-4473-ad01-99525c7b95e1-kubelet-config\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616708 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f202c1ae-315f-4f19-b0a3-706a278e1177-multus-daemon-config\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616732 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-kubelet\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616758 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn87m\" (UniqueName: \"kubernetes.io/projected/362a0176-19e5-42d8-808c-e32e578f7527-kube-api-access-bn87m\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616783 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616808 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-multus-conf-dir\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616833 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/362a0176-19e5-42d8-808c-e32e578f7527-env-overrides\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616858 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwdtr\" (UniqueName: \"kubernetes.io/projected/9704c18f-5f2d-417b-b49f-bf27e2774eef-kube-api-access-kwdtr\") pod \"network-metrics-daemon-65bkg\" (UID: \"9704c18f-5f2d-417b-b49f-bf27e2774eef\") " pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616892 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkb4\" (UniqueName: \"kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4\") pod \"network-check-target-kqtz5\" (UID: \"481c8a32-3270-4720-97fe-49de4a613919\") " pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616917 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27f20e9c-87e3-4473-ad01-99525c7b95e1-dbus\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616945 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-system-cni-dir\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616970 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-var-lib-kubelet\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.616996 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-systemd-units\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617023 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f202c1ae-315f-4f19-b0a3-706a278e1177-cni-binary-copy\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617049 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-log-socket\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617075 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617123 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-etc-kubernetes\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.617268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617158 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-run-ovn-kubernetes\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617184 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs\") pod \"network-metrics-daemon-65bkg\" (UID: \"9704c18f-5f2d-417b-b49f-bf27e2774eef\") " pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617208 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-var-lib-cni-bin\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617235 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-run-multus-certs\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617260 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-run-netns\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617285 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-run-systemd\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617313 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0c4cd2b-33ca-47fe-8263-dab2a28569cc-host\") pod \"node-ca-hfg9n\" (UID: \"f0c4cd2b-33ca-47fe-8263-dab2a28569cc\") " pod="openshift-image-registry/node-ca-hfg9n" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617339 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-cnibin\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617364 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-var-lib-openvswitch\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617390 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-multus-socket-dir-parent\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617415 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-hostroot\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617446 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-node-log\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617474 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-var-lib-cni-multus\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617498 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-slash\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617598 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-slash\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617648 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-run-openvswitch\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.618003 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.617985 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-run-ovn\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.618790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.618137 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-kubelet\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.618790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.618208 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-multus-cni-dir\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.618790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.618274 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/362a0176-19e5-42d8-808c-e32e578f7527-ovnkube-config\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.618790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.618314 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c831c7ba-6c28-4355-a3e1-271edff8c1ad-konnectivity-ca\") pod \"konnectivity-agent-4lmvp\" (UID: \"c831c7ba-6c28-4355-a3e1-271edff8c1ad\") " pod="kube-system/konnectivity-agent-4lmvp" Jul 22 16:40:37.618790 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:37.618520 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:37.618790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.618565 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/27f20e9c-87e3-4473-ad01-99525c7b95e1-dbus\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:37.618790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.618629 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-system-cni-dir\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.618790 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:37.618639 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret podName:27f20e9c-87e3-4473-ad01-99525c7b95e1 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:38.118614844 +0000 UTC m=+3.095343170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret") pod "global-pull-secret-syncer-4842j" (UID: "27f20e9c-87e3-4473-ad01-99525c7b95e1") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:37.618790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.618685 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-systemd-units\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.618790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.618686 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-multus-conf-dir\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.618790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.618720 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-var-lib-kubelet\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.618790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.618764 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-cni-netd\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.619398 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.618817 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-os-release\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.619398 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.618875 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-run-netns\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.619398 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.618917 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-cni-bin\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.619398 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.618995 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-run-netns\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.619398 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619131 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0c4cd2b-33ca-47fe-8263-dab2a28569cc-serviceca\") pod \"node-ca-hfg9n\" (UID: \"f0c4cd2b-33ca-47fe-8263-dab2a28569cc\") " pod="openshift-image-registry/node-ca-hfg9n" Jul 22 16:40:37.619398 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619196 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-run-systemd\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.619398 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619244 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0c4cd2b-33ca-47fe-8263-dab2a28569cc-host\") pod \"node-ca-hfg9n\" (UID: \"f0c4cd2b-33ca-47fe-8263-dab2a28569cc\") " pod="openshift-image-registry/node-ca-hfg9n" Jul 22 16:40:37.619398 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619253 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/362a0176-19e5-42d8-808c-e32e578f7527-env-overrides\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.619398 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619299 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-cnibin\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.619398 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619344 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-var-lib-openvswitch\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.619398 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619399 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-multus-socket-dir-parent\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619439 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-hostroot\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619492 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-var-lib-cni-multus\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619498 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-node-log\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619545 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-etc-openvswitch\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619559 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/27f20e9c-87e3-4473-ad01-99525c7b95e1-kubelet-config\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619615 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-run-ovn-kubernetes\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619623 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f202c1ae-315f-4f19-b0a3-706a278e1177-cni-binary-copy\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619663 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-log-socket\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619710 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/362a0176-19e5-42d8-808c-e32e578f7527-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:37.619716 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:37.619783 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs podName:9704c18f-5f2d-417b-b49f-bf27e2774eef nodeName:}" failed. No retries permitted until 2026-07-22 16:40:38.119766173 +0000 UTC m=+3.096494484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs") pod "network-metrics-daemon-65bkg" (UID: "9704c18f-5f2d-417b-b49f-bf27e2774eef") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619776 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-var-lib-cni-bin\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619825 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-etc-kubernetes\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619834 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-run-multus-certs\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619871 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f202c1ae-315f-4f19-b0a3-706a278e1177-host-run-k8s-cni-cncf-io\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.619997 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.619966 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f202c1ae-315f-4f19-b0a3-706a278e1177-multus-daemon-config\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.620739 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.620287 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/362a0176-19e5-42d8-808c-e32e578f7527-ovnkube-script-lib\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.622469 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.622399 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/362a0176-19e5-42d8-808c-e32e578f7527-ovn-node-metrics-cert\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.622823 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.622799 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c831c7ba-6c28-4355-a3e1-271edff8c1ad-agent-certs\") pod \"konnectivity-agent-4lmvp\" (UID: \"c831c7ba-6c28-4355-a3e1-271edff8c1ad\") " pod="kube-system/konnectivity-agent-4lmvp" Jul 22 16:40:37.628763 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:37.628419 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:40:37.628763 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:37.628448 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:40:37.628763 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:37.628465 2557 projected.go:196] Error preparing data for projected volume kube-api-access-nrkb4 for pod openshift-network-diagnostics/network-check-target-kqtz5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:37.628763 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:37.628531 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4 podName:481c8a32-3270-4720-97fe-49de4a613919 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:38.128514096 +0000 UTC m=+3.105242393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nrkb4" (UniqueName: "kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4") pod "network-check-target-kqtz5" (UID: "481c8a32-3270-4720-97fe-49de4a613919") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:37.631055 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.631011 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-889s9\" (UniqueName: \"kubernetes.io/projected/f0c4cd2b-33ca-47fe-8263-dab2a28569cc-kube-api-access-889s9\") pod \"node-ca-hfg9n\" (UID: \"f0c4cd2b-33ca-47fe-8263-dab2a28569cc\") " pod="openshift-image-registry/node-ca-hfg9n" Jul 22 16:40:37.631300 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.631278 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwdtr\" (UniqueName: \"kubernetes.io/projected/9704c18f-5f2d-417b-b49f-bf27e2774eef-kube-api-access-kwdtr\") pod \"network-metrics-daemon-65bkg\" (UID: \"9704c18f-5f2d-417b-b49f-bf27e2774eef\") " pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:37.631477 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.631420 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:40:37.631953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.631885 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l5pp\" (UniqueName: \"kubernetes.io/projected/f202c1ae-315f-4f19-b0a3-706a278e1177-kube-api-access-8l5pp\") pod \"multus-qb6fj\" (UID: \"f202c1ae-315f-4f19-b0a3-706a278e1177\") " pod="openshift-multus/multus-qb6fj" Jul 22 16:40:37.631953 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.631890 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn87m\" (UniqueName: \"kubernetes.io/projected/362a0176-19e5-42d8-808c-e32e578f7527-kube-api-access-bn87m\") pod \"ovnkube-node-2bs4s\" (UID: \"362a0176-19e5-42d8-808c-e32e578f7527\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.694977 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.694944 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 16:40:37.720840 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.720809 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ncfqd" Jul 22 16:40:37.724085 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.723916 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wjjfr" Jul 22 16:40:37.735096 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.735075 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" Jul 22 16:40:37.743205 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.743187 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kngsc" Jul 22 16:40:37.751028 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.751004 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f7wc2" Jul 22 16:40:37.759115 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.759075 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hfg9n" Jul 22 16:40:37.767012 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.766992 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4lmvp" Jul 22 16:40:37.775810 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.775786 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:40:37.783726 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:37.783709 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qb6fj" Jul 22 16:40:38.122805 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:38.122728 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs\") pod \"network-metrics-daemon-65bkg\" (UID: \"9704c18f-5f2d-417b-b49f-bf27e2774eef\") " pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:38.122805 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:38.122782 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:38.123026 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:38.122891 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:38.123026 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:38.122917 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:38.123026 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:38.122957 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret podName:27f20e9c-87e3-4473-ad01-99525c7b95e1 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:39.122940817 +0000 UTC m=+4.099669131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret") pod "global-pull-secret-syncer-4842j" (UID: "27f20e9c-87e3-4473-ad01-99525c7b95e1") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:38.123026 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:38.122981 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs podName:9704c18f-5f2d-417b-b49f-bf27e2774eef nodeName:}" failed. No retries permitted until 2026-07-22 16:40:39.122966374 +0000 UTC m=+4.099694669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs") pod "network-metrics-daemon-65bkg" (UID: "9704c18f-5f2d-417b-b49f-bf27e2774eef") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:38.223836 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:38.223793 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkb4\" (UniqueName: \"kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4\") pod \"network-check-target-kqtz5\" (UID: \"481c8a32-3270-4720-97fe-49de4a613919\") " pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:38.224019 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:38.223955 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:40:38.224019 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:38.223987 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:40:38.224019 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:38.224002 2557 projected.go:196] Error preparing data for projected volume kube-api-access-nrkb4 for pod openshift-network-diagnostics/network-check-target-kqtz5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:38.224164 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:38.224087 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4 podName:481c8a32-3270-4720-97fe-49de4a613919 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:39.224071492 +0000 UTC m=+4.200799787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrkb4" (UniqueName: "kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4") pod "network-check-target-kqtz5" (UID: "481c8a32-3270-4720-97fe-49de4a613919") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:38.448388 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:38.448306 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 16:35:38 +0000 UTC" deadline="2027-12-21 03:52:06.454851553 +0000 UTC" Jul 22 16:40:38.448388 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:38.448350 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12395h11m28.006509948s" Jul 22 16:40:38.569493 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:38.569453 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:38.569672 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:38.569579 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:40:38.814987 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:38.814949 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6acdc380_0278_49e9_a977_f16754ea4d48.slice/crio-a07f261f71214267e37f929c5c1986c61ee1d61e587557dccc9ffa4778d4ee8c WatchSource:0}: Error finding container a07f261f71214267e37f929c5c1986c61ee1d61e587557dccc9ffa4778d4ee8c: Status 404 returned error can't find the container with id a07f261f71214267e37f929c5c1986c61ee1d61e587557dccc9ffa4778d4ee8c Jul 22 16:40:38.816337 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:38.816307 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf202c1ae_315f_4f19_b0a3_706a278e1177.slice/crio-b7191fcb335a37e644b931ae4eb6880d70d94c03c3b9c9d2c2bfc097993ee715 WatchSource:0}: Error finding container b7191fcb335a37e644b931ae4eb6880d70d94c03c3b9c9d2c2bfc097993ee715: Status 404 returned error can't find the container with id b7191fcb335a37e644b931ae4eb6880d70d94c03c3b9c9d2c2bfc097993ee715 Jul 22 16:40:38.817337 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:38.817314 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7163961_e9d7_4963_804c_f906fa9f8817.slice/crio-bc2ee82b46588d81e8d31a68c4743c83b483bb686d23d7ab8ab97429159f0dfe WatchSource:0}: Error finding container bc2ee82b46588d81e8d31a68c4743c83b483bb686d23d7ab8ab97429159f0dfe: Status 404 returned error can't find the container with id bc2ee82b46588d81e8d31a68c4743c83b483bb686d23d7ab8ab97429159f0dfe Jul 22 16:40:38.819203 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:38.818476 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362a0176_19e5_42d8_808c_e32e578f7527.slice/crio-02702b0b32fc5627555703ea058c0c3d89812332e0a634bd1b97564362170a33 WatchSource:0}: Error finding container 02702b0b32fc5627555703ea058c0c3d89812332e0a634bd1b97564362170a33: Status 404 returned error can't find the container with id 02702b0b32fc5627555703ea058c0c3d89812332e0a634bd1b97564362170a33 Jul 22 16:40:38.830321 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:40:38.830276 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e4e5c9_7183_4cc7_9ea2_17ea2e4d8f97.slice/crio-19499cfec59bb07eb2542f678c0c2a04d166f3d86958fb35d7e39bb55df8785d WatchSource:0}: Error finding container 19499cfec59bb07eb2542f678c0c2a04d166f3d86958fb35d7e39bb55df8785d: Status 404 returned error can't find the container with id 19499cfec59bb07eb2542f678c0c2a04d166f3d86958fb35d7e39bb55df8785d Jul 22 16:40:39.131973 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.131677 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:39.131973 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.131960 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs\") pod \"network-metrics-daemon-65bkg\" (UID: \"9704c18f-5f2d-417b-b49f-bf27e2774eef\") " pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:39.132193 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:39.131834 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:39.132193 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:39.132081 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret podName:27f20e9c-87e3-4473-ad01-99525c7b95e1 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:41.132058689 +0000 UTC m=+6.108787003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret") pod "global-pull-secret-syncer-4842j" (UID: "27f20e9c-87e3-4473-ad01-99525c7b95e1") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:39.132193 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:39.132089 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:39.132193 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:39.132127 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs podName:9704c18f-5f2d-417b-b49f-bf27e2774eef nodeName:}" failed. No retries permitted until 2026-07-22 16:40:41.132115764 +0000 UTC m=+6.108844060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs") pod "network-metrics-daemon-65bkg" (UID: "9704c18f-5f2d-417b-b49f-bf27e2774eef") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:39.233561 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.233528 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkb4\" (UniqueName: \"kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4\") pod \"network-check-target-kqtz5\" (UID: \"481c8a32-3270-4720-97fe-49de4a613919\") " pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:39.233707 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:39.233667 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:40:39.233707 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:39.233682 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:40:39.233707 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:39.233691 2557 projected.go:196] Error preparing data for projected volume kube-api-access-nrkb4 for pod openshift-network-diagnostics/network-check-target-kqtz5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:39.233815 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:39.233745 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4 podName:481c8a32-3270-4720-97fe-49de4a613919 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:41.233726566 +0000 UTC m=+6.210454861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrkb4" (UniqueName: "kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4") pod "network-check-target-kqtz5" (UID: "481c8a32-3270-4720-97fe-49de4a613919") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:39.449401 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.449279 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 16:35:38 +0000 UTC" deadline="2028-04-12 15:20:21.292798208 +0000 UTC" Jul 22 16:40:39.449401 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.449350 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15118h39m41.843453398s" Jul 22 16:40:39.570235 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.570206 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:39.570393 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:39.570348 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:40:39.570465 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.570397 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:39.570565 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:39.570534 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:40:39.584190 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.584085 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hfg9n" event={"ID":"f0c4cd2b-33ca-47fe-8263-dab2a28569cc","Type":"ContainerStarted","Data":"9e5b1b5c873ac3d291ec9fc4202d5aca6fccafefc42dbc79796f77ffe60680b1"} Jul 22 16:40:39.586982 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.586869 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" event={"ID":"362a0176-19e5-42d8-808c-e32e578f7527","Type":"ContainerStarted","Data":"02702b0b32fc5627555703ea058c0c3d89812332e0a634bd1b97564362170a33"} Jul 22 16:40:39.590283 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.590197 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f7wc2" event={"ID":"6acdc380-0278-49e9-a977-f16754ea4d48","Type":"ContainerStarted","Data":"a07f261f71214267e37f929c5c1986c61ee1d61e587557dccc9ffa4778d4ee8c"} Jul 22 16:40:39.592807 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.592728 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kngsc" event={"ID":"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97","Type":"ContainerStarted","Data":"19499cfec59bb07eb2542f678c0c2a04d166f3d86958fb35d7e39bb55df8785d"} Jul 22 16:40:39.596090 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.596018 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4lmvp" event={"ID":"c831c7ba-6c28-4355-a3e1-271edff8c1ad","Type":"ContainerStarted","Data":"3d332d775294d96935edf183f522cfb9c415807757126abf3d4465895b6060d2"} Jul 22 16:40:39.598179 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.598006 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wjjfr" event={"ID":"c7163961-e9d7-4963-804c-f906fa9f8817","Type":"ContainerStarted","Data":"bc2ee82b46588d81e8d31a68c4743c83b483bb686d23d7ab8ab97429159f0dfe"} Jul 22 16:40:39.602353 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.602272 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qb6fj" event={"ID":"f202c1ae-315f-4f19-b0a3-706a278e1177","Type":"ContainerStarted","Data":"b7191fcb335a37e644b931ae4eb6880d70d94c03c3b9c9d2c2bfc097993ee715"} Jul 22 16:40:39.614601 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.613711 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-94.ec2.internal" event={"ID":"466327f919d9276f8f0590ca18d52c6c","Type":"ContainerStarted","Data":"8d6ba22880cec0fd9bbb27d4f808d397af04acee9b20d79ddc4593635386b909"} Jul 22 16:40:39.620540 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.620193 2557 generic.go:356] "Generic (PLEG): container finished" podID="174c6948cb8527f74edd0b4217d9cd0d" containerID="dd0c88aa2b6705330f55e6a0f64faa28daf8bb05aef13c31115b30a5c1d6c28b" exitCode=0 Jul 22 16:40:39.620540 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.620274 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal" event={"ID":"174c6948cb8527f74edd0b4217d9cd0d","Type":"ContainerDied","Data":"dd0c88aa2b6705330f55e6a0f64faa28daf8bb05aef13c31115b30a5c1d6c28b"} Jul 22 16:40:39.628176 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.628137 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncfqd" event={"ID":"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6","Type":"ContainerStarted","Data":"a42ba71d8e7640045b4cc3c9591cb4a039ff087e84bb3ca6c264468e7f7cddba"} Jul 22 16:40:39.632462 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.632377 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" event={"ID":"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0","Type":"ContainerStarted","Data":"a15793937f4c17f0d8e249b344e6abd3b3e1a3d6f2581cb33d6d92e2b5f72686"} Jul 22 16:40:39.647255 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:39.647198 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-94.ec2.internal" podStartSLOduration=2.64717846 podStartE2EDuration="2.64717846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:39.632140959 +0000 UTC m=+4.608869281" watchObservedRunningTime="2026-07-22 16:40:39.64717846 +0000 UTC m=+4.623906781" Jul 22 16:40:40.569261 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:40.569215 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:40.569788 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:40.569381 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:40:40.663056 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:40.663017 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal" event={"ID":"174c6948cb8527f74edd0b4217d9cd0d","Type":"ContainerStarted","Data":"d0eafc582d9ec8d7b63897b7337a2c69edb60defdcfd5917931e62b7966c3e9a"} Jul 22 16:40:41.154122 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:41.154083 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:41.154314 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:41.154160 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs\") pod \"network-metrics-daemon-65bkg\" (UID: \"9704c18f-5f2d-417b-b49f-bf27e2774eef\") " pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:41.154314 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:41.154213 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:41.154314 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:41.154274 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:41.154314 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:41.154283 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret podName:27f20e9c-87e3-4473-ad01-99525c7b95e1 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:45.154263219 +0000 UTC m=+10.130991538 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret") pod "global-pull-secret-syncer-4842j" (UID: "27f20e9c-87e3-4473-ad01-99525c7b95e1") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:41.154314 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:41.154309 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs podName:9704c18f-5f2d-417b-b49f-bf27e2774eef nodeName:}" failed. No retries permitted until 2026-07-22 16:40:45.154297764 +0000 UTC m=+10.131026061 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs") pod "network-metrics-daemon-65bkg" (UID: "9704c18f-5f2d-417b-b49f-bf27e2774eef") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:41.255135 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:41.254699 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkb4\" (UniqueName: \"kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4\") pod \"network-check-target-kqtz5\" (UID: \"481c8a32-3270-4720-97fe-49de4a613919\") " pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:41.255135 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:41.254969 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:40:41.255135 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:41.255014 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:40:41.255135 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:41.255029 2557 projected.go:196] Error preparing data for projected volume kube-api-access-nrkb4 for pod openshift-network-diagnostics/network-check-target-kqtz5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:41.255135 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:41.255141 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4 podName:481c8a32-3270-4720-97fe-49de4a613919 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:45.255117573 +0000 UTC m=+10.231845885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrkb4" (UniqueName: "kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4") pod "network-check-target-kqtz5" (UID: "481c8a32-3270-4720-97fe-49de4a613919") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:41.570190 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:41.569252 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:41.570190 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:41.569395 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:40:41.570190 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:41.569797 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:41.570190 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:41.569901 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:40:42.569771 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:42.569731 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:42.569974 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:42.569896 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:40:43.573401 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:43.572944 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:43.573401 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:43.573048 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:40:43.573401 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:43.573178 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:43.573401 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:43.573283 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:40:44.569474 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:44.569425 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:44.569680 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:44.569597 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:40:45.191526 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:45.191295 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:45.191526 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:45.191373 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs\") pod \"network-metrics-daemon-65bkg\" (UID: \"9704c18f-5f2d-417b-b49f-bf27e2774eef\") " pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:45.191526 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:45.191441 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:45.191526 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:45.191481 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:45.191526 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:45.191508 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret podName:27f20e9c-87e3-4473-ad01-99525c7b95e1 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:53.191488858 +0000 UTC m=+18.168217175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret") pod "global-pull-secret-syncer-4842j" (UID: "27f20e9c-87e3-4473-ad01-99525c7b95e1") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:45.191526 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:45.191527 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs podName:9704c18f-5f2d-417b-b49f-bf27e2774eef nodeName:}" failed. No retries permitted until 2026-07-22 16:40:53.191517593 +0000 UTC m=+18.168245889 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs") pod "network-metrics-daemon-65bkg" (UID: "9704c18f-5f2d-417b-b49f-bf27e2774eef") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:45.292108 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:45.292076 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkb4\" (UniqueName: \"kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4\") pod \"network-check-target-kqtz5\" (UID: \"481c8a32-3270-4720-97fe-49de4a613919\") " pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:45.292306 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:45.292285 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:40:45.292367 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:45.292317 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:40:45.292367 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:45.292331 2557 projected.go:196] Error preparing data for projected volume kube-api-access-nrkb4 for pod openshift-network-diagnostics/network-check-target-kqtz5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:45.292469 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:45.292399 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4 podName:481c8a32-3270-4720-97fe-49de4a613919 nodeName:}" failed. No retries permitted until 2026-07-22 16:40:53.292378493 +0000 UTC m=+18.269106804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrkb4" (UniqueName: "kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4") pod "network-check-target-kqtz5" (UID: "481c8a32-3270-4720-97fe-49de4a613919") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:45.570510 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:45.570419 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:45.570680 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:45.570552 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:40:45.570972 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:45.570951 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:45.571107 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:45.571057 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:40:46.569242 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:46.569110 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:46.569242 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:46.569216 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:40:47.568938 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:47.568896 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:47.569144 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:47.569058 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:40:47.569144 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:47.569095 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:47.569331 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:47.569212 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:40:48.569291 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:48.569253 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:48.569479 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:48.569393 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:40:49.569819 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:49.569750 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:49.570337 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:49.569753 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:49.570337 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:49.569885 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:40:49.570337 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:49.569947 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:40:50.569385 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:50.569300 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:50.569548 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:50.569429 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:40:51.569797 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:51.569759 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:51.570312 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:51.569895 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:40:51.570312 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:51.569947 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:51.570312 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:51.570076 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:40:52.569453 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:52.569415 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:52.569714 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:52.569551 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:40:53.254260 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:53.254224 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs\") pod \"network-metrics-daemon-65bkg\" (UID: \"9704c18f-5f2d-417b-b49f-bf27e2774eef\") " pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:53.254714 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:53.254292 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:53.254714 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:53.254397 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:53.254714 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:53.254468 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs podName:9704c18f-5f2d-417b-b49f-bf27e2774eef nodeName:}" failed. No retries permitted until 2026-07-22 16:41:09.254448607 +0000 UTC m=+34.231176904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs") pod "network-metrics-daemon-65bkg" (UID: "9704c18f-5f2d-417b-b49f-bf27e2774eef") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:40:53.254714 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:53.254408 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:53.254714 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:53.254554 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret podName:27f20e9c-87e3-4473-ad01-99525c7b95e1 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:09.254523522 +0000 UTC m=+34.231251830 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret") pod "global-pull-secret-syncer-4842j" (UID: "27f20e9c-87e3-4473-ad01-99525c7b95e1") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:40:53.355617 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:53.355583 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkb4\" (UniqueName: \"kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4\") pod \"network-check-target-kqtz5\" (UID: \"481c8a32-3270-4720-97fe-49de4a613919\") " pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:53.355821 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:53.355758 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:40:53.355821 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:53.355787 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:40:53.355821 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:53.355801 2557 projected.go:196] Error preparing data for projected volume kube-api-access-nrkb4 for pod openshift-network-diagnostics/network-check-target-kqtz5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:53.355994 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:53.355864 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4 podName:481c8a32-3270-4720-97fe-49de4a613919 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:09.355846006 +0000 UTC m=+34.332574303 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrkb4" (UniqueName: "kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4") pod "network-check-target-kqtz5" (UID: "481c8a32-3270-4720-97fe-49de4a613919") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:40:53.569327 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:53.569225 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:53.569501 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:53.569369 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:40:53.569501 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:53.569420 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:53.569611 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:53.569532 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:40:54.569451 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:54.569417 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:54.569833 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:54.569534 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:40:55.570696 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:55.570660 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:55.571199 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:55.570770 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:40:55.571199 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:55.570827 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:55.571199 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:55.570959 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:40:56.569128 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:56.569086 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:56.569311 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:56.569206 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:40:57.571706 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:57.571679 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:57.572185 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:57.571797 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:40:57.572185 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:57.571870 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:57.572185 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:57.571964 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:40:58.569458 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.569176 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:40:58.569562 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:58.569543 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:40:58.701427 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.701398 2557 generic.go:356] "Generic (PLEG): container finished" podID="ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6" containerID="12efc7b755b78909420908344605ca0931e720e302d8bad8c25818e53c901890" exitCode=0 Jul 22 16:40:58.702120 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.701498 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncfqd" event={"ID":"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6","Type":"ContainerDied","Data":"12efc7b755b78909420908344605ca0931e720e302d8bad8c25818e53c901890"} Jul 22 16:40:58.703093 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.703063 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" event={"ID":"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0","Type":"ContainerStarted","Data":"4f9d0bf8a8a76410b39c10ed6161d6a5beae23721b5e7f776eaeda67b697ed44"} Jul 22 16:40:58.704349 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.704324 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hfg9n" event={"ID":"f0c4cd2b-33ca-47fe-8263-dab2a28569cc","Type":"ContainerStarted","Data":"6ec9bb15f683fd38b9ec4dbd87d36808543df81ced7b1ec7f86597b8a9d6672c"} Jul 22 16:40:58.706835 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.706798 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" event={"ID":"362a0176-19e5-42d8-808c-e32e578f7527","Type":"ContainerStarted","Data":"9f1b3d79dac7a1c50b2791d3c5997f943a2a9db9c1cdb2e3811807c329ae35c9"} Jul 22 16:40:58.706941 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.706852 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" event={"ID":"362a0176-19e5-42d8-808c-e32e578f7527","Type":"ContainerStarted","Data":"f867b91bf6723c7122d7b628b315052aa6fba1958ae533f594258a49ee88d467"} Jul 22 16:40:58.706941 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.706883 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" event={"ID":"362a0176-19e5-42d8-808c-e32e578f7527","Type":"ContainerStarted","Data":"1311faf497705612d9697f5f2d0ba821ca8d39e84f994d37068810f298fbe7c2"} Jul 22 16:40:58.706941 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.706910 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" event={"ID":"362a0176-19e5-42d8-808c-e32e578f7527","Type":"ContainerStarted","Data":"8c319067e2e42bd47338bd4fcd50b7208f4983ff81b10b0d8ff8c8327a13173a"} Jul 22 16:40:58.706941 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.706936 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" event={"ID":"362a0176-19e5-42d8-808c-e32e578f7527","Type":"ContainerStarted","Data":"7d4f4899005473b436bded30fe7753b3d4b476377970be4b9e0a952c84241ceb"} Jul 22 16:40:58.708260 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.708238 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f7wc2" event={"ID":"6acdc380-0278-49e9-a977-f16754ea4d48","Type":"ContainerStarted","Data":"c0404170df2226c55b650fd23e5d3227d6bea3daa3826f158ca58741513272ec"} Jul 22 16:40:58.709747 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.709636 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kngsc" event={"ID":"52e4e5c9-7183-4cc7-9ea2-17ea2e4d8f97","Type":"ContainerStarted","Data":"78937c53da0588f3004ce4e06454f82280d8fa3105c94dd2d180fcb2f80ff1c5"} Jul 22 16:40:58.710989 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.710969 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4lmvp" event={"ID":"c831c7ba-6c28-4355-a3e1-271edff8c1ad","Type":"ContainerStarted","Data":"ffaf2f63912b6c45527c0642d83d5b03ff6c4045d7cc29798f3547685b0143d4"} Jul 22 16:40:58.712293 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.712275 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qb6fj" event={"ID":"f202c1ae-315f-4f19-b0a3-706a278e1177","Type":"ContainerStarted","Data":"b83c7c66691e0d0cf883bc8ecb99993e956fbbf38d8d72ce5ef4f20d0e5d0f0c"} Jul 22 16:40:58.726666 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.726218 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-94.ec2.internal" podStartSLOduration=21.726200912 podStartE2EDuration="21.726200912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:40.682161198 +0000 UTC m=+5.658889517" watchObservedRunningTime="2026-07-22 16:40:58.726200912 +0000 UTC m=+23.702929232" Jul 22 16:40:58.740294 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.740257 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f7wc2" podStartSLOduration=3.350297297 podStartE2EDuration="22.740244508s" totalImagesPullingTime="19.389947211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:58.740141393 +0000 UTC m=+23.716869709" watchObservedRunningTime="2026-07-22 16:40:58.740244508 +0000 UTC m=+23.716972824" Jul 22 16:40:58.776493 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.776454 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hfg9n" podStartSLOduration=3.395038798 podStartE2EDuration="22.77644169s" totalImagesPullingTime="19.381402892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:58.77616071 +0000 UTC m=+23.752889025" watchObservedRunningTime="2026-07-22 16:40:58.77644169 +0000 UTC m=+23.753170006" Jul 22 16:40:58.776609 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.776591 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4lmvp" podStartSLOduration=7.057717593 podStartE2EDuration="22.776584363s" totalImagesPullingTime="15.71886677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:58.756751834 +0000 UTC m=+23.733480151" watchObservedRunningTime="2026-07-22 16:40:58.776584363 +0000 UTC m=+23.753312678" Jul 22 16:40:58.823269 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.823228 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kngsc" podStartSLOduration=3.429572184 podStartE2EDuration="22.823215609s" totalImagesPullingTime="19.393643425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:58.823187954 +0000 UTC m=+23.799916283" watchObservedRunningTime="2026-07-22 16:40:58.823215609 +0000 UTC m=+23.799943925" Jul 22 16:40:58.823469 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.823443 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qb6fj" podStartSLOduration=3.368075115 podStartE2EDuration="22.823436736s" totalImagesPullingTime="19.455361621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:58.794875777 +0000 UTC m=+23.771604093" watchObservedRunningTime="2026-07-22 16:40:58.823436736 +0000 UTC m=+23.800165052" Jul 22 16:40:58.895984 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.895944 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:58.969603 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:58.969527 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:59.070449 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:59.070410 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:59.518212 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:59.518114 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:59.572995 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:59.572967 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:40:59.573187 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:59.573086 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:40:59.573187 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:59.573147 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:40:59.573301 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:40:59.573258 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:40:59.608620 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:59.608589 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:59.677626 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:59.677584 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:59.717437 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:59.717404 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" event={"ID":"362a0176-19e5-42d8-808c-e32e578f7527","Type":"ContainerStarted","Data":"af38280c67bef35d7dcc9c122da689cb937ae3cdbb4519b86d61b759a4d54462"} Jul 22 16:40:59.719113 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:59.719074 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wjjfr" event={"ID":"c7163961-e9d7-4963-804c-f906fa9f8817","Type":"ContainerStarted","Data":"c70ecd0dbd98ab3e105e6925cf9023671f9abb7100691fc9cc38160398794474"} Jul 22 16:40:59.748011 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:59.747955 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wjjfr" podStartSLOduration=4.353233707 podStartE2EDuration="23.747935332s" totalImagesPullingTime="19.394701625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:40:59.747465327 +0000 UTC m=+24.724193856" watchObservedRunningTime="2026-07-22 16:40:59.747935332 +0000 UTC m=+24.724663652" Jul 22 16:40:59.767097 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:59.767065 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:59.840037 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:59.839996 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:59.926326 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:59.926301 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:40:59.971685 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:59.971658 2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 16:40:59.994143 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:40:59.994115 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:00.266336 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:00.266246 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:00.365589 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:00.365554 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:00.477090 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:00.476987 2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T16:40:59.971679403Z","UUID":"37dacbf8-c8ff-4cef-a0a0-613315834ec0","Handler":null,"Name":"","Endpoint":""} Jul 22 16:41:00.478600 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:00.478565 2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 16:41:00.478600 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:00.478606 2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 16:41:00.569268 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:00.569192 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:00.569406 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:00.569310 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:00.723517 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:00.723471 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" event={"ID":"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0","Type":"ContainerStarted","Data":"83af33f78298ec302d22cbd2ee7f4d58285e200a415fc4ade28dca4c5df95f6d"} Jul 22 16:41:01.569286 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:01.569257 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:01.569507 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:01.569257 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:01.569507 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:01.569379 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:01.569507 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:01.569446 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:01.728405 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:01.728322 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" event={"ID":"7766a7cd-d6fc-4331-8a4b-2a21ba8eeba0","Type":"ContainerStarted","Data":"6ba73670c9eb5a2656f145bc181c9e5735d7f4ad18bf0638084211458f765880"} Jul 22 16:41:01.732125 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:01.732092 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" event={"ID":"362a0176-19e5-42d8-808c-e32e578f7527","Type":"ContainerStarted","Data":"489240b0d6fc9d4483dbb6f6833af311c8c835f9e416b0c0852c524b240a0d94"} Jul 22 16:41:01.873179 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:01.873142 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:01.973384 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:01.973345 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:02.149309 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:02.149240 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f7wc2_6acdc380-0278-49e9-a977-f16754ea4d48/dns-node-resolver/0.log" Jul 22 16:41:02.569056 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:02.568786 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:02.569234 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:02.569137 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:02.746076 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:02.746049 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hfg9n_f0c4cd2b-33ca-47fe-8263-dab2a28569cc/node-ca/0.log" Jul 22 16:41:03.340971 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:03.340799 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4lmvp" Jul 22 16:41:03.341625 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:03.341497 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4lmvp" Jul 22 16:41:03.360991 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:03.360949 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pcgp7" podStartSLOduration=5.199036716 podStartE2EDuration="27.3609367s" totalImagesPullingTime="22.161899984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:01.749116149 +0000 UTC m=+26.725844464" watchObservedRunningTime="2026-07-22 16:41:03.3609367 +0000 UTC m=+28.337665016" Jul 22 16:41:03.420506 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:03.420467 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:03.569888 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:03.569646 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:03.569888 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:03.569661 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:03.570158 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:03.569893 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:03.570158 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:03.569966 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:03.737910 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:03.737874 2557 generic.go:356] "Generic (PLEG): container finished" podID="ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6" containerID="6ffc46ef335e3e47882dd16ff18bcb1dfe027c675421c43a6b106b16273287d8" exitCode=0 Jul 22 16:41:03.738110 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:03.737955 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncfqd" event={"ID":"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6","Type":"ContainerDied","Data":"6ffc46ef335e3e47882dd16ff18bcb1dfe027c675421c43a6b106b16273287d8"} Jul 22 16:41:03.741767 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:03.741644 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" event={"ID":"362a0176-19e5-42d8-808c-e32e578f7527","Type":"ContainerStarted","Data":"a18defb900a18072bf1a31d84b12069eeab808315c66fda13edd11740debfbc4"} Jul 22 16:41:03.742051 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:03.742019 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4lmvp" Jul 22 16:41:03.742292 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:03.742279 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4lmvp" Jul 22 16:41:03.817862 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:03.817809 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" podStartSLOduration=8.354422013 podStartE2EDuration="27.817793635s" totalImagesPullingTime="19.463371622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:03.816795372 +0000 UTC m=+28.793523688" watchObservedRunningTime="2026-07-22 16:41:03.817793635 +0000 UTC m=+28.794521969" Jul 22 16:41:04.569686 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:04.569655 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:04.569866 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:04.569761 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:04.743589 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:04.743557 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:41:04.743757 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:04.743630 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:41:04.743757 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:04.743671 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:41:04.758779 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:04.758753 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:41:04.758931 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:04.758853 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:41:05.406187 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:05.406154 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:05.570082 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:05.570051 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:05.570228 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:05.570130 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:05.570228 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:05.570188 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:05.570297 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:05.570275 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:05.747402 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:05.747317 2557 generic.go:356] "Generic (PLEG): container finished" podID="ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6" containerID="d430e5e9fb71577ebc1aa4e4e2d6db4c1968ba08de7b21042e257682446f207c" exitCode=0 Jul 22 16:41:05.747402 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:05.747393 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncfqd" event={"ID":"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6","Type":"ContainerDied","Data":"d430e5e9fb71577ebc1aa4e4e2d6db4c1968ba08de7b21042e257682446f207c"} Jul 22 16:41:06.568698 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:06.568668 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:06.569090 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:06.568775 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:06.600668 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:06.600637 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:06.681368 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:06.681317 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:07.301373 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:07.301336 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:07.410568 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:07.410537 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:07.512336 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:07.512307 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:07.569396 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:07.569321 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:07.569727 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:07.569445 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:07.569727 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:07.569495 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:07.569727 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:07.569582 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:07.753571 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:07.753533 2557 generic.go:356] "Generic (PLEG): container finished" podID="ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6" containerID="38a480767f1aedbba554de6989f1e8bc50f9b85bd629f9cd44e2e0345e9f8551" exitCode=0 Jul 22 16:41:07.753724 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:07.753592 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncfqd" event={"ID":"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6","Type":"ContainerDied","Data":"38a480767f1aedbba554de6989f1e8bc50f9b85bd629f9cd44e2e0345e9f8551"} Jul 22 16:41:08.185549 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:08.185512 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:08.287252 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:08.287214 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:08.569532 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:08.569503 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:08.569925 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:08.569628 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:09.141893 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:09.141850 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:09.240149 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:09.240116 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:09.277188 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:09.277152 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:09.277378 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:09.277220 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs\") pod \"network-metrics-daemon-65bkg\" (UID: \"9704c18f-5f2d-417b-b49f-bf27e2774eef\") " pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:09.277378 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:09.277282 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 16:41:09.277378 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:09.277339 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:41:09.277378 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:09.277354 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret podName:27f20e9c-87e3-4473-ad01-99525c7b95e1 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:41.277334739 +0000 UTC m=+66.254063055 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret") pod "global-pull-secret-syncer-4842j" (UID: "27f20e9c-87e3-4473-ad01-99525c7b95e1") : object "kube-system"/"original-pull-secret" not registered Jul 22 16:41:09.277558 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:09.277390 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs podName:9704c18f-5f2d-417b-b49f-bf27e2774eef nodeName:}" failed. No retries permitted until 2026-07-22 16:41:41.277372478 +0000 UTC m=+66.254100772 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs") pod "network-metrics-daemon-65bkg" (UID: "9704c18f-5f2d-417b-b49f-bf27e2774eef") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 16:41:09.340281 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:09.340240 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:09.378343 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:09.378313 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkb4\" (UniqueName: \"kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4\") pod \"network-check-target-kqtz5\" (UID: \"481c8a32-3270-4720-97fe-49de4a613919\") " pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:09.378533 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:09.378469 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 16:41:09.378533 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:09.378490 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 16:41:09.378533 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:09.378502 2557 projected.go:196] Error preparing data for projected volume kube-api-access-nrkb4 for pod openshift-network-diagnostics/network-check-target-kqtz5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:41:09.378693 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:09.378565 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4 podName:481c8a32-3270-4720-97fe-49de4a613919 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:41.378545544 +0000 UTC m=+66.355273856 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrkb4" (UniqueName: "kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4") pod "network-check-target-kqtz5" (UID: "481c8a32-3270-4720-97fe-49de4a613919") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 16:41:09.495796 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:09.495761 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:09.568839 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:09.568800 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:09.568839 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:09.568804 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:09.569084 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:09.568954 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:09.569084 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:09.569044 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:09.690037 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:09.689987 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:09.892735 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:09.892650 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:10.095903 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:10.095815 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:10.569335 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:10.569296 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:10.569508 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:10.569418 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:10.894285 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:10.894205 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:11.083794 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:11.083747 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:11.148683 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:11.148591 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:11.490049 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:11.489994 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:11.569062 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:11.569006 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:11.569062 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:11.569063 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:11.569298 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:11.569160 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:11.569298 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:11.569274 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:11.693522 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:11.693479 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:11.759843 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:11.759758 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:11.890394 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:11.890354 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:12.114087 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:12.114057 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:12.569432 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:12.569397 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:12.569594 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:12.569504 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:12.769067 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:12.769017 2557 generic.go:356] "Generic (PLEG): container finished" podID="ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6" containerID="2af790a159458365c086b6f57d0fce30c4582696fd8cf9341dd68a3ea6fa5a5a" exitCode=0 Jul 22 16:41:12.769235 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:12.769097 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncfqd" event={"ID":"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6","Type":"ContainerDied","Data":"2af790a159458365c086b6f57d0fce30c4582696fd8cf9341dd68a3ea6fa5a5a"} Jul 22 16:41:13.290457 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:13.290426 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:13.498800 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:13.498764 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:13.569827 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:13.569745 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:13.569968 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:13.569874 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:13.569968 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:13.569919 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:13.570080 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:13.569998 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:13.693222 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:13.693182 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:13.769943 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:13.769903 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:13.773817 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:13.773787 2557 generic.go:356] "Generic (PLEG): container finished" podID="ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6" containerID="f3704ca73212b291ce894f51c321dac2fbe2c514981be18fe92ff15ab185107c" exitCode=0 Jul 22 16:41:13.773946 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:13.773847 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncfqd" event={"ID":"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6","Type":"ContainerDied","Data":"f3704ca73212b291ce894f51c321dac2fbe2c514981be18fe92ff15ab185107c"} Jul 22 16:41:13.889640 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:13.889460 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:14.089002 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:14.088919 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:14.314242 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:14.314208 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:14.569053 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:14.569012 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:14.569211 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:14.569128 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:14.779275 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:14.779246 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncfqd" event={"ID":"ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6","Type":"ContainerStarted","Data":"612cf79228624902aae5df4ab832a9a1697cc7159a1599b242f422559d241607"} Jul 22 16:41:14.805517 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:14.805470 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ncfqd" podStartSLOduration=8.897505684 podStartE2EDuration="38.805458085s" totalImagesPullingTime="29.907952401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:14.804927646 +0000 UTC m=+39.781655964" watchObservedRunningTime="2026-07-22 16:41:14.805458085 +0000 UTC m=+39.782186401" Jul 22 16:41:15.090890 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:15.090856 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:15.493382 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:15.493351 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:15.570707 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:15.570676 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:15.570842 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:15.570801 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:15.570885 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:15.570838 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:15.570943 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:15.570926 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:15.686979 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:15.686949 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:16.013425 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:16.013381 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:16.294635 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:16.294543 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:16.568915 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:16.568832 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:16.569337 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:16.568979 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:16.888764 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:16.888673 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:17.504303 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:17.504273 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:17.568881 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:17.568841 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:17.569074 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:17.568858 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:17.569074 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:17.568959 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:17.569074 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:17.569018 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:17.687252 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:17.687218 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:17.889308 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:17.889226 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:18.085044 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:18.084998 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:18.569236 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:18.569198 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:18.569678 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:18.569389 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:19.444632 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:19.444597 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:19.569475 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:19.569444 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:19.569902 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:19.569548 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:19.569902 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:19.569611 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:19.569902 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:19.569736 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:20.569259 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:20.569228 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:20.569427 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:20.569329 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:21.569676 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:21.569646 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:21.569676 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:21.569659 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:21.570239 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:21.569764 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:21.570239 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:21.569873 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:22.568859 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:22.568828 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:22.569044 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:22.568934 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:23.569362 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:23.569325 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:23.569731 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:23.569463 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:23.569731 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:23.569510 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:23.569731 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:23.569592 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:24.115661 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:24.115632 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:24.222100 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:24.222069 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:24.569637 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:24.569600 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:24.570119 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:24.569728 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:25.570270 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:25.570238 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:25.570736 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:25.570313 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:25.570736 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:25.570418 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:25.570736 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:25.570507 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:26.569134 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:26.569104 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:26.569297 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:26.569207 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:27.569090 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:27.569058 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:27.569467 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:27.569157 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:27.569467 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:27.569214 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:27.569467 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:27.569299 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:27.605562 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:27.605533 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:27.706146 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:27.706113 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:27.812854 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:27.812818 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:27.915386 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:27.915303 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:28.008319 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:28.008274 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:28.082811 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:28.082784 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:28.163109 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:28.163071 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:28.569464 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:28.569432 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:28.569881 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:28.569567 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:29.568765 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:29.568721 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:29.568765 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:29.568721 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:29.568989 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:29.568839 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:29.569080 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:29.569057 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:29.584741 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:29.584714 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-65bkg"] Jul 22 16:41:29.585373 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:29.584798 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:29.585373 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:29.584878 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:29.586742 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:29.586721 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4842j"] Jul 22 16:41:29.591324 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:29.591255 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kqtz5"] Jul 22 16:41:29.809703 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:29.809673 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:29.809876 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:29.809677 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:29.809876 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:29.809767 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:29.809876 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:29.809867 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:30.310264 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:30.310161 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:30.413761 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:30.413725 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:30.485385 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:30.485340 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:30.548204 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:30.548173 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:31.569527 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:31.569500 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:31.570080 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:31.569604 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:31.570080 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:31.569500 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:31.570080 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:31.569642 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:31.570080 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:31.569710 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:31.570080 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:31.569769 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:32.170778 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:32.170746 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:32.242225 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:32.242193 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:32.561703 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:32.561670 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:33.569232 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.569203 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:33.569642 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.569206 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:33.569642 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:33.569309 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kqtz5" podUID="481c8a32-3270-4720-97fe-49de4a613919" Jul 22 16:41:33.569642 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:33.569388 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4842j" podUID="27f20e9c-87e3-4473-ad01-99525c7b95e1" Jul 22 16:41:33.569642 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.569432 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:33.569642 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:33.569512 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-65bkg" podUID="9704c18f-5f2d-417b-b49f-bf27e2774eef" Jul 22 16:41:33.824314 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.824241 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-94.ec2.internal" event="NodeReady" Jul 22 16:41:33.824502 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.824375 2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 16:41:33.887271 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.887238 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gbnnc"] Jul 22 16:41:33.890111 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.890094 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:33.893138 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.893111 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 16:41:33.893304 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.893292 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 16:41:33.893346 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.893308 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n8ggb\"" Jul 22 16:41:33.907761 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.907732 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d9zkt"] Jul 22 16:41:33.910577 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.910558 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:33.913010 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.912989 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gbnnc"] Jul 22 16:41:33.913576 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.913555 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hs6zh\"" Jul 22 16:41:33.914019 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.913994 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 16:41:33.914142 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.914089 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 16:41:33.914142 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.914116 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 16:41:33.914394 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.914377 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 16:41:33.927052 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.926326 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d9zkt"] Jul 22 16:41:33.950626 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.950587 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e46da55-c83c-410b-95d5-e6d5af057a64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9zkt\" (UID: \"2e46da55-c83c-410b-95d5-e6d5af057a64\") " pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:33.950785 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.950640 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e46da55-c83c-410b-95d5-e6d5af057a64-data-volume\") pod \"insights-runtime-extractor-d9zkt\" (UID: \"2e46da55-c83c-410b-95d5-e6d5af057a64\") " pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:33.950785 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.950700 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c6bfe9-79f1-4d51-bc5f-04d11054953e-config-volume\") pod \"dns-default-gbnnc\" (UID: \"f4c6bfe9-79f1-4d51-bc5f-04d11054953e\") " pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:33.950785 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.950737 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7mdh\" (UniqueName: \"kubernetes.io/projected/f4c6bfe9-79f1-4d51-bc5f-04d11054953e-kube-api-access-q7mdh\") pod \"dns-default-gbnnc\" (UID: \"f4c6bfe9-79f1-4d51-bc5f-04d11054953e\") " pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:33.950785 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.950772 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm2px\" (UniqueName: \"kubernetes.io/projected/2e46da55-c83c-410b-95d5-e6d5af057a64-kube-api-access-nm2px\") pod \"insights-runtime-extractor-d9zkt\" (UID: \"2e46da55-c83c-410b-95d5-e6d5af057a64\") " pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:33.950913 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.950788 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4c6bfe9-79f1-4d51-bc5f-04d11054953e-metrics-tls\") pod \"dns-default-gbnnc\" (UID: \"f4c6bfe9-79f1-4d51-bc5f-04d11054953e\") " pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:33.950913 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.950824 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c6bfe9-79f1-4d51-bc5f-04d11054953e-tmp-dir\") pod \"dns-default-gbnnc\" (UID: \"f4c6bfe9-79f1-4d51-bc5f-04d11054953e\") " pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:33.950974 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.950926 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e46da55-c83c-410b-95d5-e6d5af057a64-crio-socket\") pod \"insights-runtime-extractor-d9zkt\" (UID: \"2e46da55-c83c-410b-95d5-e6d5af057a64\") " pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:33.950974 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.950944 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e46da55-c83c-410b-95d5-e6d5af057a64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9zkt\" (UID: \"2e46da55-c83c-410b-95d5-e6d5af057a64\") " pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:33.970651 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.970618 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g5wlp"] Jul 22 16:41:33.973445 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.973430 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g5wlp" Jul 22 16:41:33.976141 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.976114 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 16:41:33.976297 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.976180 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mkm8j\"" Jul 22 16:41:33.976297 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.976226 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 16:41:33.976297 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.976277 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 16:41:33.983557 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:33.983535 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g5wlp"] Jul 22 16:41:34.046556 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.046509 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:34.051457 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.051437 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e46da55-c83c-410b-95d5-e6d5af057a64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9zkt\" (UID: \"2e46da55-c83c-410b-95d5-e6d5af057a64\") " pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:34.051551 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.051469 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e46da55-c83c-410b-95d5-e6d5af057a64-data-volume\") pod \"insights-runtime-extractor-d9zkt\" (UID: \"2e46da55-c83c-410b-95d5-e6d5af057a64\") " pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:34.051551 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.051492 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c6bfe9-79f1-4d51-bc5f-04d11054953e-config-volume\") pod \"dns-default-gbnnc\" (UID: \"f4c6bfe9-79f1-4d51-bc5f-04d11054953e\") " pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:34.051551 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.051514 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7mdh\" (UniqueName: \"kubernetes.io/projected/f4c6bfe9-79f1-4d51-bc5f-04d11054953e-kube-api-access-q7mdh\") pod \"dns-default-gbnnc\" (UID: \"f4c6bfe9-79f1-4d51-bc5f-04d11054953e\") " pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:34.051551 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.051542 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm2px\" (UniqueName: \"kubernetes.io/projected/2e46da55-c83c-410b-95d5-e6d5af057a64-kube-api-access-nm2px\") pod \"insights-runtime-extractor-d9zkt\" (UID: \"2e46da55-c83c-410b-95d5-e6d5af057a64\") " pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:34.051694 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.051565 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4c6bfe9-79f1-4d51-bc5f-04d11054953e-metrics-tls\") pod \"dns-default-gbnnc\" (UID: \"f4c6bfe9-79f1-4d51-bc5f-04d11054953e\") " pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:34.051694 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.051588 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c6bfe9-79f1-4d51-bc5f-04d11054953e-tmp-dir\") pod \"dns-default-gbnnc\" (UID: \"f4c6bfe9-79f1-4d51-bc5f-04d11054953e\") " pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:34.051694 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.051650 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm22l\" (UniqueName: \"kubernetes.io/projected/f58b7e81-5b73-4f25-8194-1869ad43e260-kube-api-access-jm22l\") pod \"ingress-canary-g5wlp\" (UID: \"f58b7e81-5b73-4f25-8194-1869ad43e260\") " pod="openshift-ingress-canary/ingress-canary-g5wlp" Jul 22 16:41:34.051694 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.051678 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f58b7e81-5b73-4f25-8194-1869ad43e260-cert\") pod \"ingress-canary-g5wlp\" (UID: \"f58b7e81-5b73-4f25-8194-1869ad43e260\") " pod="openshift-ingress-canary/ingress-canary-g5wlp" Jul 22 16:41:34.051835 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.051726 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e46da55-c83c-410b-95d5-e6d5af057a64-crio-socket\") pod \"insights-runtime-extractor-d9zkt\" (UID: \"2e46da55-c83c-410b-95d5-e6d5af057a64\") " pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:34.051835 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.051753 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e46da55-c83c-410b-95d5-e6d5af057a64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9zkt\" (UID: \"2e46da55-c83c-410b-95d5-e6d5af057a64\") " pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:34.052015 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.051861 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e46da55-c83c-410b-95d5-e6d5af057a64-data-volume\") pod \"insights-runtime-extractor-d9zkt\" (UID: \"2e46da55-c83c-410b-95d5-e6d5af057a64\") " pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:34.052141 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.052116 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e46da55-c83c-410b-95d5-e6d5af057a64-crio-socket\") pod \"insights-runtime-extractor-d9zkt\" (UID: \"2e46da55-c83c-410b-95d5-e6d5af057a64\") " pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:34.052234 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.052136 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4c6bfe9-79f1-4d51-bc5f-04d11054953e-tmp-dir\") pod \"dns-default-gbnnc\" (UID: \"f4c6bfe9-79f1-4d51-bc5f-04d11054953e\") " pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:34.052234 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.052213 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c6bfe9-79f1-4d51-bc5f-04d11054953e-config-volume\") pod \"dns-default-gbnnc\" (UID: \"f4c6bfe9-79f1-4d51-bc5f-04d11054953e\") " pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:34.052312 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.052285 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e46da55-c83c-410b-95d5-e6d5af057a64-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d9zkt\" (UID: \"2e46da55-c83c-410b-95d5-e6d5af057a64\") " pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:34.055652 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.055631 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4c6bfe9-79f1-4d51-bc5f-04d11054953e-metrics-tls\") pod \"dns-default-gbnnc\" (UID: \"f4c6bfe9-79f1-4d51-bc5f-04d11054953e\") " pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:34.055768 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.055749 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e46da55-c83c-410b-95d5-e6d5af057a64-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d9zkt\" (UID: \"2e46da55-c83c-410b-95d5-e6d5af057a64\") " pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:34.061463 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.061441 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7mdh\" (UniqueName: \"kubernetes.io/projected/f4c6bfe9-79f1-4d51-bc5f-04d11054953e-kube-api-access-q7mdh\") pod \"dns-default-gbnnc\" (UID: \"f4c6bfe9-79f1-4d51-bc5f-04d11054953e\") " pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:34.061559 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.061531 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm2px\" (UniqueName: \"kubernetes.io/projected/2e46da55-c83c-410b-95d5-e6d5af057a64-kube-api-access-nm2px\") pod \"insights-runtime-extractor-d9zkt\" (UID: \"2e46da55-c83c-410b-95d5-e6d5af057a64\") " pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:34.151669 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.151577 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:34.152401 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.152370 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm22l\" (UniqueName: \"kubernetes.io/projected/f58b7e81-5b73-4f25-8194-1869ad43e260-kube-api-access-jm22l\") pod \"ingress-canary-g5wlp\" (UID: \"f58b7e81-5b73-4f25-8194-1869ad43e260\") " pod="openshift-ingress-canary/ingress-canary-g5wlp" Jul 22 16:41:34.152502 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.152407 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f58b7e81-5b73-4f25-8194-1869ad43e260-cert\") pod \"ingress-canary-g5wlp\" (UID: \"f58b7e81-5b73-4f25-8194-1869ad43e260\") " pod="openshift-ingress-canary/ingress-canary-g5wlp" Jul 22 16:41:34.154757 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.154735 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f58b7e81-5b73-4f25-8194-1869ad43e260-cert\") pod \"ingress-canary-g5wlp\" (UID: \"f58b7e81-5b73-4f25-8194-1869ad43e260\") " pod="openshift-ingress-canary/ingress-canary-g5wlp" Jul 22 16:41:34.160881 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.160858 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm22l\" (UniqueName: \"kubernetes.io/projected/f58b7e81-5b73-4f25-8194-1869ad43e260-kube-api-access-jm22l\") pod \"ingress-canary-g5wlp\" (UID: \"f58b7e81-5b73-4f25-8194-1869ad43e260\") " pod="openshift-ingress-canary/ingress-canary-g5wlp" Jul 22 16:41:34.199751 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.199713 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:34.219965 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.219941 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d9zkt" Jul 22 16:41:34.281739 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.281701 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g5wlp" Jul 22 16:41:34.344558 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.341484 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gbnnc"] Jul 22 16:41:34.352758 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:41:34.352708 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c6bfe9_79f1_4d51_bc5f_04d11054953e.slice/crio-73e698e2a654647453cbf2d1cfa055e1689daebd24f82cafc9e3bb44464c0fc0 WatchSource:0}: Error finding container 73e698e2a654647453cbf2d1cfa055e1689daebd24f82cafc9e3bb44464c0fc0: Status 404 returned error can't find the container with id 73e698e2a654647453cbf2d1cfa055e1689daebd24f82cafc9e3bb44464c0fc0 Jul 22 16:41:34.355554 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.355354 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d9zkt"] Jul 22 16:41:34.372217 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:41:34.372191 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e46da55_c83c_410b_95d5_e6d5af057a64.slice/crio-554e0730ebd17acae27ac07d89985709c3d6438e4d85dad9280fa4d3db917bad WatchSource:0}: Error finding container 554e0730ebd17acae27ac07d89985709c3d6438e4d85dad9280fa4d3db917bad: Status 404 returned error can't find the container with id 554e0730ebd17acae27ac07d89985709c3d6438e4d85dad9280fa4d3db917bad Jul 22 16:41:34.428107 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.427998 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g5wlp"] Jul 22 16:41:34.430877 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:41:34.430851 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58b7e81_5b73_4f25_8194_1869ad43e260.slice/crio-e66bdc0691c34827abdae24bf8a7c9e9576007d35da99c3143ea76923f09eb43 WatchSource:0}: Error finding container e66bdc0691c34827abdae24bf8a7c9e9576007d35da99c3143ea76923f09eb43: Status 404 returned error can't find the container with id e66bdc0691c34827abdae24bf8a7c9e9576007d35da99c3143ea76923f09eb43 Jul 22 16:41:34.515538 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.515502 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:34.548735 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.548702 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc"] Jul 22 16:41:34.552833 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.552817 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" Jul 22 16:41:34.555426 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.555403 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 16:41:34.555536 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.555470 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 22 16:41:34.555536 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.555505 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-96xlh\"" Jul 22 16:41:34.555536 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.555513 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 16:41:34.555754 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.555734 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 22 16:41:34.555796 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.555769 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 16:41:34.565487 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.565469 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc"] Jul 22 16:41:34.588829 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.588793 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:34.655699 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.655672 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4487fd14-d796-4638-8f4d-731fad568b10-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-pfnpc\" (UID: \"4487fd14-d796-4638-8f4d-731fad568b10\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" Jul 22 16:41:34.655855 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.655705 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4487fd14-d796-4638-8f4d-731fad568b10-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-pfnpc\" (UID: \"4487fd14-d796-4638-8f4d-731fad568b10\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" Jul 22 16:41:34.655855 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.655780 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkxrv\" (UniqueName: \"kubernetes.io/projected/4487fd14-d796-4638-8f4d-731fad568b10-kube-api-access-qkxrv\") pod \"prometheus-operator-5945b5cfd-pfnpc\" (UID: \"4487fd14-d796-4638-8f4d-731fad568b10\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" Jul 22 16:41:34.655855 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.655817 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4487fd14-d796-4638-8f4d-731fad568b10-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-pfnpc\" (UID: \"4487fd14-d796-4638-8f4d-731fad568b10\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" Jul 22 16:41:34.757215 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.756725 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4487fd14-d796-4638-8f4d-731fad568b10-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-pfnpc\" (UID: \"4487fd14-d796-4638-8f4d-731fad568b10\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" Jul 22 16:41:34.757215 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.756771 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4487fd14-d796-4638-8f4d-731fad568b10-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-pfnpc\" (UID: \"4487fd14-d796-4638-8f4d-731fad568b10\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" Jul 22 16:41:34.757215 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.756833 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkxrv\" (UniqueName: \"kubernetes.io/projected/4487fd14-d796-4638-8f4d-731fad568b10-kube-api-access-qkxrv\") pod \"prometheus-operator-5945b5cfd-pfnpc\" (UID: \"4487fd14-d796-4638-8f4d-731fad568b10\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" Jul 22 16:41:34.757215 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.756889 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4487fd14-d796-4638-8f4d-731fad568b10-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-pfnpc\" (UID: \"4487fd14-d796-4638-8f4d-731fad568b10\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" Jul 22 16:41:34.757215 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:34.757008 2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 22 16:41:34.757215 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:34.757102 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4487fd14-d796-4638-8f4d-731fad568b10-prometheus-operator-tls podName:4487fd14-d796-4638-8f4d-731fad568b10 nodeName:}" failed. No retries permitted until 2026-07-22 16:41:35.257083443 +0000 UTC m=+60.233811760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/4487fd14-d796-4638-8f4d-731fad568b10-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-pfnpc" (UID: "4487fd14-d796-4638-8f4d-731fad568b10") : secret "prometheus-operator-tls" not found Jul 22 16:41:34.757994 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.757968 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4487fd14-d796-4638-8f4d-731fad568b10-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-pfnpc\" (UID: \"4487fd14-d796-4638-8f4d-731fad568b10\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" Jul 22 16:41:34.760187 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.760163 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4487fd14-d796-4638-8f4d-731fad568b10-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-pfnpc\" (UID: \"4487fd14-d796-4638-8f4d-731fad568b10\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" Jul 22 16:41:34.769175 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.769151 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkxrv\" (UniqueName: \"kubernetes.io/projected/4487fd14-d796-4638-8f4d-731fad568b10-kube-api-access-qkxrv\") pod \"prometheus-operator-5945b5cfd-pfnpc\" (UID: \"4487fd14-d796-4638-8f4d-731fad568b10\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" Jul 22 16:41:34.824148 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.824106 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d9zkt" event={"ID":"2e46da55-c83c-410b-95d5-e6d5af057a64","Type":"ContainerStarted","Data":"554e0730ebd17acae27ac07d89985709c3d6438e4d85dad9280fa4d3db917bad"} Jul 22 16:41:34.825904 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.825870 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g5wlp" event={"ID":"f58b7e81-5b73-4f25-8194-1869ad43e260","Type":"ContainerStarted","Data":"e66bdc0691c34827abdae24bf8a7c9e9576007d35da99c3143ea76923f09eb43"} Jul 22 16:41:34.827468 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.827389 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gbnnc" event={"ID":"f4c6bfe9-79f1-4d51-bc5f-04d11054953e","Type":"ContainerStarted","Data":"73e698e2a654647453cbf2d1cfa055e1689daebd24f82cafc9e3bb44464c0fc0"} Jul 22 16:41:34.897942 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:34.897909 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:35.261358 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:35.261316 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4487fd14-d796-4638-8f4d-731fad568b10-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-pfnpc\" (UID: \"4487fd14-d796-4638-8f4d-731fad568b10\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" Jul 22 16:41:35.268150 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:35.268094 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4487fd14-d796-4638-8f4d-731fad568b10-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-pfnpc\" (UID: \"4487fd14-d796-4638-8f4d-731fad568b10\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" Jul 22 16:41:35.464480 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:35.464256 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-96xlh\"" Jul 22 16:41:35.472432 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:35.472399 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" Jul 22 16:41:35.570401 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:35.570369 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:35.570864 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:35.570762 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:35.571214 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:35.571197 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:35.575055 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:35.575013 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 16:41:35.575788 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:35.575765 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vs4sl\"" Jul 22 16:41:35.575902 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:35.575839 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 16:41:35.576218 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:35.576007 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 16:41:35.576218 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:35.576070 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 16:41:35.576218 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:35.576156 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d5r55\"" Jul 22 16:41:35.642642 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:35.642610 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc"] Jul 22 16:41:35.647146 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:41:35.647102 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4487fd14_d796_4638_8f4d_731fad568b10.slice/crio-10920e7910224413ba45404df41be20722269424c125f0d747a3636327d7f907 WatchSource:0}: Error finding container 10920e7910224413ba45404df41be20722269424c125f0d747a3636327d7f907: Status 404 returned error can't find the container with id 10920e7910224413ba45404df41be20722269424c125f0d747a3636327d7f907 Jul 22 16:41:35.832666 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:35.832588 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" event={"ID":"4487fd14-d796-4638-8f4d-731fad568b10","Type":"ContainerStarted","Data":"10920e7910224413ba45404df41be20722269424c125f0d747a3636327d7f907"} Jul 22 16:41:36.763591 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:36.763515 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2bs4s" Jul 22 16:41:36.839177 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:36.839138 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gbnnc" event={"ID":"f4c6bfe9-79f1-4d51-bc5f-04d11054953e","Type":"ContainerStarted","Data":"65e6fc33f3938be72c1abae366733839cd5b054cd79649115292f22d0c54dd51"} Jul 22 16:41:36.839351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:36.839202 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gbnnc" event={"ID":"f4c6bfe9-79f1-4d51-bc5f-04d11054953e","Type":"ContainerStarted","Data":"d44c11e3bad0a9c417863aa10ffca08732e9d04dcca35d264f6bff5a53dbdbe7"} Jul 22 16:41:36.839891 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:36.839869 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:36.858517 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:36.858449 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gbnnc" podStartSLOduration=1.722506163 podStartE2EDuration="3.858432461s" totalImagesPullingTime="2.135926298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:36.857901352 +0000 UTC m=+61.834629670" watchObservedRunningTime="2026-07-22 16:41:36.858432461 +0000 UTC m=+61.835160777" Jul 22 16:41:36.882162 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:36.882121 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:36.939069 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:36.939001 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:37.002982 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:37.002919 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:40.853978 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:40.853450 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" event={"ID":"4487fd14-d796-4638-8f4d-731fad568b10","Type":"ContainerStarted","Data":"47823688cd850babfaece843e15465fee923bdd436c15647430fb0495a6d1c33"} Jul 22 16:41:40.853978 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:40.853520 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" event={"ID":"4487fd14-d796-4638-8f4d-731fad568b10","Type":"ContainerStarted","Data":"36c2efc4c7f88d1596dcfb89cd3d6969f886686ee8d3b498c861f75123f05533"} Jul 22 16:41:40.856577 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:40.856545 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d9zkt" event={"ID":"2e46da55-c83c-410b-95d5-e6d5af057a64","Type":"ContainerStarted","Data":"6489fec567f77ad789b430f777e61d5480c387b0fabd51cb4b5e64871a3aed67"} Jul 22 16:41:40.858333 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:40.858304 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g5wlp" event={"ID":"f58b7e81-5b73-4f25-8194-1869ad43e260","Type":"ContainerStarted","Data":"5f2966aba466695dfd053e1713340f7a4fe6b94e2a6a4f80e1905835798e5bd1"} Jul 22 16:41:40.873455 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:40.873398 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-pfnpc" podStartSLOduration=1.820770634 podStartE2EDuration="6.873382825s" totalImagesPullingTime="5.052612191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:40.872460216 +0000 UTC m=+65.849188533" watchObservedRunningTime="2026-07-22 16:41:40.873382825 +0000 UTC m=+65.850111140" Jul 22 16:41:41.036780 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.036744 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:41.313470 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.313372 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:41.313608 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.313478 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs\") pod \"network-metrics-daemon-65bkg\" (UID: \"9704c18f-5f2d-417b-b49f-bf27e2774eef\") " pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:41.315999 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.315974 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/27f20e9c-87e3-4473-ad01-99525c7b95e1-original-pull-secret\") pod \"global-pull-secret-syncer-4842j\" (UID: \"27f20e9c-87e3-4473-ad01-99525c7b95e1\") " pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:41.316129 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.316053 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9704c18f-5f2d-417b-b49f-bf27e2774eef-metrics-certs\") pod \"network-metrics-daemon-65bkg\" (UID: \"9704c18f-5f2d-417b-b49f-bf27e2774eef\") " pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:41.414242 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.414206 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkb4\" (UniqueName: \"kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4\") pod \"network-check-target-kqtz5\" (UID: \"481c8a32-3270-4720-97fe-49de4a613919\") " pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:41.417340 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.417289 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkb4\" (UniqueName: \"kubernetes.io/projected/481c8a32-3270-4720-97fe-49de4a613919-kube-api-access-nrkb4\") pod \"network-check-target-kqtz5\" (UID: \"481c8a32-3270-4720-97fe-49de4a613919\") " pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:41.586995 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.586914 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:41.594995 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.594967 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4842j" Jul 22 16:41:41.602080 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.602052 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-65bkg" Jul 22 16:41:41.761540 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.760854 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g5wlp" podStartSLOduration=2.494357275 podStartE2EDuration="8.760828968s" totalImagesPullingTime="6.266471693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:40.892749013 +0000 UTC m=+65.869477330" watchObservedRunningTime="2026-07-22 16:41:41.760828968 +0000 UTC m=+66.737557285" Jul 22 16:41:41.761802 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.761594 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kqtz5"] Jul 22 16:41:41.766630 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:41:41.766600 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481c8a32_3270_4720_97fe_49de4a613919.slice/crio-95692628d348db933d6bd034e302161537e861088e80242b249f681c620a1040 WatchSource:0}: Error finding container 95692628d348db933d6bd034e302161537e861088e80242b249f681c620a1040: Status 404 returned error can't find the container with id 95692628d348db933d6bd034e302161537e861088e80242b249f681c620a1040 Jul 22 16:41:41.782218 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.782190 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4842j"] Jul 22 16:41:41.785569 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:41:41.785538 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f20e9c_87e3_4473_ad01_99525c7b95e1.slice/crio-859a91a53bb0e33b696ffe76edd8b3cdcacec6732687f9d2e5b16cb175fc7247 WatchSource:0}: Error finding container 859a91a53bb0e33b696ffe76edd8b3cdcacec6732687f9d2e5b16cb175fc7247: Status 404 returned error can't find the container with id 859a91a53bb0e33b696ffe76edd8b3cdcacec6732687f9d2e5b16cb175fc7247 Jul 22 16:41:41.798940 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.798916 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-65bkg"] Jul 22 16:41:41.802536 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:41:41.802507 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9704c18f_5f2d_417b_b49f_bf27e2774eef.slice/crio-c9032ade4e0d7fb0a37092d8359654421bbdd8d4ac1e5f7897dfdd8e4e046d01 WatchSource:0}: Error finding container c9032ade4e0d7fb0a37092d8359654421bbdd8d4ac1e5f7897dfdd8e4e046d01: Status 404 returned error can't find the container with id c9032ade4e0d7fb0a37092d8359654421bbdd8d4ac1e5f7897dfdd8e4e046d01 Jul 22 16:41:41.863189 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.863114 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-65bkg" event={"ID":"9704c18f-5f2d-417b-b49f-bf27e2774eef","Type":"ContainerStarted","Data":"c9032ade4e0d7fb0a37092d8359654421bbdd8d4ac1e5f7897dfdd8e4e046d01"} Jul 22 16:41:41.864440 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.864406 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4842j" event={"ID":"27f20e9c-87e3-4473-ad01-99525c7b95e1","Type":"ContainerStarted","Data":"859a91a53bb0e33b696ffe76edd8b3cdcacec6732687f9d2e5b16cb175fc7247"} Jul 22 16:41:41.865567 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:41.865544 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kqtz5" event={"ID":"481c8a32-3270-4720-97fe-49de4a613919","Type":"ContainerStarted","Data":"95692628d348db933d6bd034e302161537e861088e80242b249f681c620a1040"} Jul 22 16:41:42.630871 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.630779 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:42.863782 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.863631 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:42.874684 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.874654 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t"] Jul 22 16:41:42.881914 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.881843 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" Jul 22 16:41:42.884699 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.884667 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7fm8p\"" Jul 22 16:41:42.884871 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.884667 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 22 16:41:42.885727 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.885686 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 22 16:41:42.890316 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.890266 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t"] Jul 22 16:41:42.911887 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.910880 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-mpf4c"] Jul 22 16:41:42.915656 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.914780 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:42.918626 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.918561 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2vd2k\"" Jul 22 16:41:42.919064 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.918917 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 22 16:41:42.919064 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.918996 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 22 16:41:42.919240 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.919148 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 22 16:41:42.924211 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.924183 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lqf5c"] Jul 22 16:41:42.927087 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.926824 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12249663-3e2d-425c-a1ed-76788ad82cbd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-wz62t\" (UID: \"12249663-3e2d-425c-a1ed-76788ad82cbd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" Jul 22 16:41:42.927087 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.926901 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12249663-3e2d-425c-a1ed-76788ad82cbd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-wz62t\" (UID: \"12249663-3e2d-425c-a1ed-76788ad82cbd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" Jul 22 16:41:42.927087 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.926933 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggrtd\" (UniqueName: \"kubernetes.io/projected/12249663-3e2d-425c-a1ed-76788ad82cbd-kube-api-access-ggrtd\") pod \"openshift-state-metrics-79f6958cdf-wz62t\" (UID: \"12249663-3e2d-425c-a1ed-76788ad82cbd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" Jul 22 16:41:42.927087 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.926965 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12249663-3e2d-425c-a1ed-76788ad82cbd-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-wz62t\" (UID: \"12249663-3e2d-425c-a1ed-76788ad82cbd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" Jul 22 16:41:42.928623 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.928004 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:42.930233 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.930176 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-mpf4c"] Jul 22 16:41:42.931041 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.930873 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 16:41:42.931041 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.930911 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 16:41:42.931544 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.931379 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 16:41:42.931544 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.931497 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qv2jv\"" Jul 22 16:41:42.940695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:42.940630 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:43.027301 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027271 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.027474 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027325 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d321f9a7-ee1f-4c09-aab2-157e849f71f3-sys\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.027474 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027347 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.027474 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027371 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.027474 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027397 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d321f9a7-ee1f-4c09-aab2-157e849f71f3-node-exporter-wtmp\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.027474 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027437 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12249663-3e2d-425c-a1ed-76788ad82cbd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-wz62t\" (UID: \"12249663-3e2d-425c-a1ed-76788ad82cbd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" Jul 22 16:41:43.027732 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027475 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d321f9a7-ee1f-4c09-aab2-157e849f71f3-node-exporter-textfile\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.027732 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027502 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfn42\" (UniqueName: \"kubernetes.io/projected/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-kube-api-access-dfn42\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.027732 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027522 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12249663-3e2d-425c-a1ed-76788ad82cbd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-wz62t\" (UID: \"12249663-3e2d-425c-a1ed-76788ad82cbd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" Jul 22 16:41:43.027732 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027544 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d321f9a7-ee1f-4c09-aab2-157e849f71f3-root\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.027732 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027569 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d321f9a7-ee1f-4c09-aab2-157e849f71f3-node-exporter-accelerators-collector-config\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.027732 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027656 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d321f9a7-ee1f-4c09-aab2-157e849f71f3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.027732 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027730 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d321f9a7-ee1f-4c09-aab2-157e849f71f3-node-exporter-tls\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.028086 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027765 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d321f9a7-ee1f-4c09-aab2-157e849f71f3-metrics-client-ca\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.028086 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027793 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhg84\" (UniqueName: \"kubernetes.io/projected/d321f9a7-ee1f-4c09-aab2-157e849f71f3-kube-api-access-zhg84\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.028086 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027855 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.028086 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027907 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggrtd\" (UniqueName: \"kubernetes.io/projected/12249663-3e2d-425c-a1ed-76788ad82cbd-kube-api-access-ggrtd\") pod \"openshift-state-metrics-79f6958cdf-wz62t\" (UID: \"12249663-3e2d-425c-a1ed-76788ad82cbd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" Jul 22 16:41:43.028086 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027940 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.028086 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.027972 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12249663-3e2d-425c-a1ed-76788ad82cbd-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-wz62t\" (UID: \"12249663-3e2d-425c-a1ed-76788ad82cbd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" Jul 22 16:41:43.028810 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.028764 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/12249663-3e2d-425c-a1ed-76788ad82cbd-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-wz62t\" (UID: \"12249663-3e2d-425c-a1ed-76788ad82cbd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" Jul 22 16:41:43.030327 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.030283 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/12249663-3e2d-425c-a1ed-76788ad82cbd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-wz62t\" (UID: \"12249663-3e2d-425c-a1ed-76788ad82cbd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" Jul 22 16:41:43.030569 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.030546 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/12249663-3e2d-425c-a1ed-76788ad82cbd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-wz62t\" (UID: \"12249663-3e2d-425c-a1ed-76788ad82cbd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" Jul 22 16:41:43.040066 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.040040 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggrtd\" (UniqueName: \"kubernetes.io/projected/12249663-3e2d-425c-a1ed-76788ad82cbd-kube-api-access-ggrtd\") pod \"openshift-state-metrics-79f6958cdf-wz62t\" (UID: \"12249663-3e2d-425c-a1ed-76788ad82cbd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" Jul 22 16:41:43.064099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.064018 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:43.128384 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128331 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d321f9a7-ee1f-4c09-aab2-157e849f71f3-root\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.128384 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128375 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d321f9a7-ee1f-4c09-aab2-157e849f71f3-node-exporter-accelerators-collector-config\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.128608 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128411 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d321f9a7-ee1f-4c09-aab2-157e849f71f3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.128608 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128432 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d321f9a7-ee1f-4c09-aab2-157e849f71f3-root\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.128608 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128445 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d321f9a7-ee1f-4c09-aab2-157e849f71f3-node-exporter-tls\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.128608 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128498 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d321f9a7-ee1f-4c09-aab2-157e849f71f3-metrics-client-ca\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.128608 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128531 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhg84\" (UniqueName: \"kubernetes.io/projected/d321f9a7-ee1f-4c09-aab2-157e849f71f3-kube-api-access-zhg84\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.128608 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128559 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.128608 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128592 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.129417 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128632 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.129417 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128665 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d321f9a7-ee1f-4c09-aab2-157e849f71f3-sys\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.129417 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128689 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.129417 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128722 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.129417 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128752 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d321f9a7-ee1f-4c09-aab2-157e849f71f3-node-exporter-wtmp\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.129417 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128794 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d321f9a7-ee1f-4c09-aab2-157e849f71f3-node-exporter-textfile\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.129417 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128818 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfn42\" (UniqueName: \"kubernetes.io/projected/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-kube-api-access-dfn42\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.129417 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.128911 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d321f9a7-ee1f-4c09-aab2-157e849f71f3-sys\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.129417 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.129290 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.129843 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.129593 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.129910 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.129881 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.130061 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.130034 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d321f9a7-ee1f-4c09-aab2-157e849f71f3-node-exporter-wtmp\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.130249 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.130230 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d321f9a7-ee1f-4c09-aab2-157e849f71f3-node-exporter-accelerators-collector-config\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.130331 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.130292 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d321f9a7-ee1f-4c09-aab2-157e849f71f3-node-exporter-textfile\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.130383 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.130339 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d321f9a7-ee1f-4c09-aab2-157e849f71f3-metrics-client-ca\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.135111 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.133489 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.135111 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.133942 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.135407 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.135385 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d321f9a7-ee1f-4c09-aab2-157e849f71f3-node-exporter-tls\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.137976 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.137932 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d321f9a7-ee1f-4c09-aab2-157e849f71f3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.140722 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.140282 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhg84\" (UniqueName: \"kubernetes.io/projected/d321f9a7-ee1f-4c09-aab2-157e849f71f3-kube-api-access-zhg84\") pod \"node-exporter-lqf5c\" (UID: \"d321f9a7-ee1f-4c09-aab2-157e849f71f3\") " pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.141094 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.141054 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfn42\" (UniqueName: \"kubernetes.io/projected/a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db-kube-api-access-dfn42\") pod \"kube-state-metrics-56c5594674-mpf4c\" (UID: \"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.184174 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.184136 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:43.195749 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.195721 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" Jul 22 16:41:43.228891 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.228853 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" Jul 22 16:41:43.240381 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.240289 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lqf5c" Jul 22 16:41:43.353498 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.353417 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:43.535337 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.535281 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:43.658716 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.658632 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:43.871672 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.871484 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:43.962857 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.962810 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:43.964454 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:43.964423 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:41:44.080146 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.080082 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:44.099398 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.099366 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:41:44.099587 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.099511 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.102476 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.102448 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 16:41:44.102603 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.102470 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 16:41:44.102603 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.102519 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 16:41:44.102603 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.102476 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 16:41:44.102765 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.102747 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 16:41:44.103158 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.102969 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 16:41:44.103158 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.102986 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 16:41:44.103158 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.102987 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 16:41:44.103158 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.103066 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-84ksk\"" Jul 22 16:41:44.108682 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.108633 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 16:41:44.239558 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.239524 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.239742 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.239573 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-config-volume\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.239742 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.239606 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.239742 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.239631 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7dc993a-dfa6-4a57-a116-87fc2a273d37-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.239742 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.239665 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7dc993a-dfa6-4a57-a116-87fc2a273d37-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.239742 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.239706 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzpvs\" (UniqueName: \"kubernetes.io/projected/d7dc993a-dfa6-4a57-a116-87fc2a273d37-kube-api-access-dzpvs\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.239742 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.239728 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-web-config\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.240060 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.239763 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7dc993a-dfa6-4a57-a116-87fc2a273d37-config-out\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.240060 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.239815 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.240060 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.239841 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7dc993a-dfa6-4a57-a116-87fc2a273d37-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.240060 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.239869 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.240060 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.239900 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7dc993a-dfa6-4a57-a116-87fc2a273d37-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.240060 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.239923 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.341036 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.340991 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.341220 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.341073 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7dc993a-dfa6-4a57-a116-87fc2a273d37-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.341220 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.341101 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.341220 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.341142 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.341220 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.341170 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-config-volume\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.341543 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.341284 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.341543 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.341330 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7dc993a-dfa6-4a57-a116-87fc2a273d37-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.341543 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.341367 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7dc993a-dfa6-4a57-a116-87fc2a273d37-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.341543 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.341413 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzpvs\" (UniqueName: \"kubernetes.io/projected/d7dc993a-dfa6-4a57-a116-87fc2a273d37-kube-api-access-dzpvs\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.341543 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.341441 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-web-config\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.341543 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.341477 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7dc993a-dfa6-4a57-a116-87fc2a273d37-config-out\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.341543 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.341532 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.341929 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.341556 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7dc993a-dfa6-4a57-a116-87fc2a273d37-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.344415 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.343942 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-config-volume\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.344415 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.344320 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.344746 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.344626 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7dc993a-dfa6-4a57-a116-87fc2a273d37-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.344746 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.344663 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7dc993a-dfa6-4a57-a116-87fc2a273d37-config-out\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.344746 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.344720 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.344982 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.344868 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7dc993a-dfa6-4a57-a116-87fc2a273d37-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.345310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.345056 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-web-config\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.345496 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.345452 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7dc993a-dfa6-4a57-a116-87fc2a273d37-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.346623 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.346565 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7dc993a-dfa6-4a57-a116-87fc2a273d37-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.346720 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.346643 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.347223 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.347196 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.347660 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.347637 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.351640 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.351617 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzpvs\" (UniqueName: \"kubernetes.io/projected/d7dc993a-dfa6-4a57-a116-87fc2a273d37-kube-api-access-dzpvs\") pod \"alertmanager-main-0\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:44.411720 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:44.411684 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:41:45.842509 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.842463 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:45.891459 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.891419 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7675958d48-7ndcw"] Jul 22 16:41:45.894219 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.894200 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:45.897772 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.897753 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-69irinfvb28u1\"" Jul 22 16:41:45.897876 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.897756 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 22 16:41:45.898255 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.898212 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 22 16:41:45.898809 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.898787 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 22 16:41:45.898809 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.898797 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 22 16:41:45.898809 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.898806 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gnw9x\"" Jul 22 16:41:45.899046 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.898787 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 22 16:41:45.910828 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.910798 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7675958d48-7ndcw"] Jul 22 16:41:45.919199 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.919176 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:45.955579 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.955547 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:45.955730 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.955587 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-grpc-tls\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:45.955730 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.955633 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86fhq\" (UniqueName: \"kubernetes.io/projected/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-kube-api-access-86fhq\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:45.955730 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.955656 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-thanos-querier-tls\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:45.955730 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.955713 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:45.955936 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.955751 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-metrics-client-ca\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:45.955936 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.955776 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:45.955936 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:45.955815 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.056874 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.056837 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86fhq\" (UniqueName: \"kubernetes.io/projected/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-kube-api-access-86fhq\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.057082 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.056886 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-thanos-querier-tls\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.057082 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.056932 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.057082 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.056972 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-metrics-client-ca\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.057082 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.057000 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.057779 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.057339 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.057779 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.057408 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.057779 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.057441 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-grpc-tls\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.058014 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.057789 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-metrics-client-ca\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.060127 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.060100 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.060369 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.060344 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.060497 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.060478 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-grpc-tls\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.060621 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.060594 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-thanos-querier-tls\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.061080 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.061016 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.061155 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.061102 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.068083 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.068059 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86fhq\" (UniqueName: \"kubernetes.io/projected/f35a1044-4f56-4b49-b1f8-73d70c28b7b3-kube-api-access-86fhq\") pod \"thanos-querier-7675958d48-7ndcw\" (UID: \"f35a1044-4f56-4b49-b1f8-73d70c28b7b3\") " pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.196670 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.196575 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:46.205079 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.205046 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:46.298616 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.298580 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:46.394801 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.394765 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:46.517536 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.517501 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:46.617622 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:46.617583 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:47.446731 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.446698 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-77c5f697f5-9ckm9"] Jul 22 16:41:47.458259 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.458225 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.462047 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.462001 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 22 16:41:47.463462 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.463438 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 22 16:41:47.463462 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.463456 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 22 16:41:47.463611 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.463482 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 22 16:41:47.463762 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.463747 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4sh508tnojrjm\"" Jul 22 16:41:47.463809 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.463760 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8n78k\"" Jul 22 16:41:47.467331 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.467305 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77c5f697f5-9ckm9"] Jul 22 16:41:47.569762 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.569726 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-secret-metrics-server-client-certs\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.569938 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.569771 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-client-ca-bundle\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.569938 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.569812 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-metrics-server-audit-profiles\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.569938 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.569887 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-secret-metrics-server-tls\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.569938 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.569931 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.570152 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.569973 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7jsh\" (UniqueName: \"kubernetes.io/projected/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-kube-api-access-c7jsh\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.570152 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.570006 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-audit-log\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.601351 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:41:47.601320 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd321f9a7_ee1f_4c09_aab2_157e849f71f3.slice/crio-e16dd40ef07d7b19546a703ac252491be7bf47bda343a3674cdfb389c5e41003 WatchSource:0}: Error finding container e16dd40ef07d7b19546a703ac252491be7bf47bda343a3674cdfb389c5e41003: Status 404 returned error can't find the container with id e16dd40ef07d7b19546a703ac252491be7bf47bda343a3674cdfb389c5e41003 Jul 22 16:41:47.671408 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.671374 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-secret-metrics-server-client-certs\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.671611 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.671424 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-client-ca-bundle\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.671611 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.671462 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-metrics-server-audit-profiles\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.671611 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.671516 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-secret-metrics-server-tls\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.671611 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.671568 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.671825 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.671615 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7jsh\" (UniqueName: \"kubernetes.io/projected/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-kube-api-access-c7jsh\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.671825 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.671653 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-audit-log\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.672100 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.672076 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-audit-log\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.675155 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.673924 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.675155 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.675103 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-metrics-server-audit-profiles\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.675331 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.675244 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-secret-metrics-server-tls\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.689292 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.689136 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:47.693130 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.692478 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc"] Jul 22 16:41:47.695542 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.694291 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-client-ca-bundle\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.695542 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.694327 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-secret-metrics-server-client-certs\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.697756 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.697223 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7jsh\" (UniqueName: \"kubernetes.io/projected/ade6b05e-41d7-43fe-b7e3-0bdc444fad5e-kube-api-access-c7jsh\") pod \"metrics-server-77c5f697f5-9ckm9\" (UID: \"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e\") " pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.698456 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.698314 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc" Jul 22 16:41:47.710766 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.706837 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc"] Jul 22 16:41:47.710766 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.707078 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wcbfb\"" Jul 22 16:41:47.710766 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.707305 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 22 16:41:47.768430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.768403 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:41:47.779412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.779169 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5d6a867-9a45-4387-983a-99649541632b-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-2ndfc\" (UID: \"a5d6a867-9a45-4387-983a-99649541632b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc" Jul 22 16:41:47.779412 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.779284 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4mth\" (UniqueName: \"kubernetes.io/projected/a5d6a867-9a45-4387-983a-99649541632b-kube-api-access-b4mth\") pod \"monitoring-plugin-7b7b7bc8bf-2ndfc\" (UID: \"a5d6a867-9a45-4387-983a-99649541632b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc" Jul 22 16:41:47.846803 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.846774 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gbnnc" Jul 22 16:41:47.881230 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.881196 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4mth\" (UniqueName: \"kubernetes.io/projected/a5d6a867-9a45-4387-983a-99649541632b-kube-api-access-b4mth\") pod \"monitoring-plugin-7b7b7bc8bf-2ndfc\" (UID: \"a5d6a867-9a45-4387-983a-99649541632b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc" Jul 22 16:41:47.881397 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.881314 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5d6a867-9a45-4387-983a-99649541632b-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-2ndfc\" (UID: \"a5d6a867-9a45-4387-983a-99649541632b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc" Jul 22 16:41:47.881514 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:47.881477 2557 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 22 16:41:47.881609 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:41:47.881589 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5d6a867-9a45-4387-983a-99649541632b-monitoring-plugin-cert podName:a5d6a867-9a45-4387-983a-99649541632b nodeName:}" failed. No retries permitted until 2026-07-22 16:41:48.381547127 +0000 UTC m=+73.358275440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a5d6a867-9a45-4387-983a-99649541632b-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-2ndfc" (UID: "a5d6a867-9a45-4387-983a-99649541632b") : secret "monitoring-plugin-cert" not found Jul 22 16:41:47.892674 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.892635 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-65bkg" event={"ID":"9704c18f-5f2d-417b-b49f-bf27e2774eef","Type":"ContainerStarted","Data":"a1b34a349b7f37c5558bc8523d1859529beb9454ed78e31bedae7e75ccb23a98"} Jul 22 16:41:47.894990 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.894954 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4842j" event={"ID":"27f20e9c-87e3-4473-ad01-99525c7b95e1","Type":"ContainerStarted","Data":"7f1698831c1fa5a50db2d9d670d95ebb83f687eaae4492bc027ef53a4aeeae0f"} Jul 22 16:41:47.896798 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.896773 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lqf5c" event={"ID":"d321f9a7-ee1f-4c09-aab2-157e849f71f3","Type":"ContainerStarted","Data":"e16dd40ef07d7b19546a703ac252491be7bf47bda343a3674cdfb389c5e41003"} Jul 22 16:41:47.903343 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.903315 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d9zkt" event={"ID":"2e46da55-c83c-410b-95d5-e6d5af057a64","Type":"ContainerStarted","Data":"64e0b813acfc5d2523e71b291fcd78b7f9e84d3b23b5d1dca929cc9fde87483b"} Jul 22 16:41:47.910910 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.910433 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kqtz5" event={"ID":"481c8a32-3270-4720-97fe-49de4a613919","Type":"ContainerStarted","Data":"debc1ba0d386e7a45c9200e4cf873375a4c7d6d8eb1043ec7d5c7c75ea23664c"} Jul 22 16:41:47.910910 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.910848 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:41:47.913444 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.913420 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7675958d48-7ndcw"] Jul 22 16:41:47.914668 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.914624 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4mth\" (UniqueName: \"kubernetes.io/projected/a5d6a867-9a45-4387-983a-99649541632b-kube-api-access-b4mth\") pod \"monitoring-plugin-7b7b7bc8bf-2ndfc\" (UID: \"a5d6a867-9a45-4387-983a-99649541632b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc" Jul 22 16:41:47.925557 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:41:47.925516 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35a1044_4f56_4b49_b1f8_73d70c28b7b3.slice/crio-97e92c69d6659b4fbd6a272925e020022662bc4b6d33e95c8358892ed1a772c7 WatchSource:0}: Error finding container 97e92c69d6659b4fbd6a272925e020022662bc4b6d33e95c8358892ed1a772c7: Status 404 returned error can't find the container with id 97e92c69d6659b4fbd6a272925e020022662bc4b6d33e95c8358892ed1a772c7 Jul 22 16:41:47.942797 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.942740 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4842j" podStartSLOduration=66.119635202 podStartE2EDuration="1m11.942717017s" totalImagesPullingTime="5.823081815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:47.94269878 +0000 UTC m=+72.919427097" watchObservedRunningTime="2026-07-22 16:41:47.942717017 +0000 UTC m=+72.919445333" Jul 22 16:41:47.969681 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.969652 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77c5f697f5-9ckm9"] Jul 22 16:41:47.972142 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:47.972098 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kqtz5" podStartSLOduration=66.226499381 podStartE2EDuration="1m11.972082907s" totalImagesPullingTime="5.745583526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:47.971215248 +0000 UTC m=+72.947943566" watchObservedRunningTime="2026-07-22 16:41:47.972082907 +0000 UTC m=+72.948811223" Jul 22 16:41:47.974114 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:41:47.974088 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade6b05e_41d7_43fe_b7e3_0bdc444fad5e.slice/crio-0c6452346519ea1e609f2ad5c9a422c216a937c1e9efb3a86c18db72ce009c14 WatchSource:0}: Error finding container 0c6452346519ea1e609f2ad5c9a422c216a937c1e9efb3a86c18db72ce009c14: Status 404 returned error can't find the container with id 0c6452346519ea1e609f2ad5c9a422c216a937c1e9efb3a86c18db72ce009c14 Jul 22 16:41:48.043862 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.043828 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t"] Jul 22 16:41:48.049193 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:41:48.049168 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12249663_3e2d_425c_a1ed_76788ad82cbd.slice/crio-229b57ffa9d8f53704ea88aa9e46867cc5f7dcc39d7d99be3049bd84befd2c98 WatchSource:0}: Error finding container 229b57ffa9d8f53704ea88aa9e46867cc5f7dcc39d7d99be3049bd84befd2c98: Status 404 returned error can't find the container with id 229b57ffa9d8f53704ea88aa9e46867cc5f7dcc39d7d99be3049bd84befd2c98 Jul 22 16:41:48.065715 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.065693 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:41:48.068511 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:41:48.068486 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7dc993a_dfa6_4a57_a116_87fc2a273d37.slice/crio-4d2aba2c1eeb766a9635e6610fb8e268ea4ebafa91d87f311849a57cdf793baa WatchSource:0}: Error finding container 4d2aba2c1eeb766a9635e6610fb8e268ea4ebafa91d87f311849a57cdf793baa: Status 404 returned error can't find the container with id 4d2aba2c1eeb766a9635e6610fb8e268ea4ebafa91d87f311849a57cdf793baa Jul 22 16:41:48.070818 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.070710 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-mpf4c"] Jul 22 16:41:48.078926 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:41:48.078899 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ed1995_b3a8_4c3c_b0a5_3ed1a1a147db.slice/crio-4b6bfbf22c0839fb53a5c72103e3e9cc38e711fff4f76c883b636068e857d9fe WatchSource:0}: Error finding container 4b6bfbf22c0839fb53a5c72103e3e9cc38e711fff4f76c883b636068e857d9fe: Status 404 returned error can't find the container with id 4b6bfbf22c0839fb53a5c72103e3e9cc38e711fff4f76c883b636068e857d9fe Jul 22 16:41:48.229630 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.229587 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:48.386001 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.385954 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5d6a867-9a45-4387-983a-99649541632b-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-2ndfc\" (UID: \"a5d6a867-9a45-4387-983a-99649541632b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc" Jul 22 16:41:48.388548 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.388524 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a5d6a867-9a45-4387-983a-99649541632b-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-2ndfc\" (UID: \"a5d6a867-9a45-4387-983a-99649541632b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc" Jul 22 16:41:48.432303 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.432267 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-phdn9"] Jul 22 16:41:48.435796 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.435764 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-phdn9" Jul 22 16:41:48.438691 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.438655 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 16:41:48.438858 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.438808 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:48.439383 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.439364 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wtxwq\"" Jul 22 16:41:48.439543 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.439384 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 16:41:48.446559 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.446513 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-phdn9"] Jul 22 16:41:48.539824 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.539663 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:48.588833 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.588803 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsf94\" (UniqueName: \"kubernetes.io/projected/19ea0909-e6e0-419a-a6cd-31bcb406be6e-kube-api-access-gsf94\") pod \"downloads-5bc67dcd46-phdn9\" (UID: \"19ea0909-e6e0-419a-a6cd-31bcb406be6e\") " pod="openshift-console/downloads-5bc67dcd46-phdn9" Jul 22 16:41:48.601479 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.601441 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:48.679162 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.679128 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc" Jul 22 16:41:48.690576 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.690162 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsf94\" (UniqueName: \"kubernetes.io/projected/19ea0909-e6e0-419a-a6cd-31bcb406be6e-kube-api-access-gsf94\") pod \"downloads-5bc67dcd46-phdn9\" (UID: \"19ea0909-e6e0-419a-a6cd-31bcb406be6e\") " pod="openshift-console/downloads-5bc67dcd46-phdn9" Jul 22 16:41:48.701267 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.701241 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsf94\" (UniqueName: \"kubernetes.io/projected/19ea0909-e6e0-419a-a6cd-31bcb406be6e-kube-api-access-gsf94\") pod \"downloads-5bc67dcd46-phdn9\" (UID: \"19ea0909-e6e0-419a-a6cd-31bcb406be6e\") " pod="openshift-console/downloads-5bc67dcd46-phdn9" Jul 22 16:41:48.733254 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.733109 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:48.749841 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.749810 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-phdn9" Jul 22 16:41:48.894595 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.894512 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc"] Jul 22 16:41:48.903680 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:41:48.903577 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d6a867_9a45_4387_983a_99649541632b.slice/crio-27f179542bb8490364d5f120e2167eb0b9f352f98dd1ff172da7866481069998 WatchSource:0}: Error finding container 27f179542bb8490364d5f120e2167eb0b9f352f98dd1ff172da7866481069998: Status 404 returned error can't find the container with id 27f179542bb8490364d5f120e2167eb0b9f352f98dd1ff172da7866481069998 Jul 22 16:41:48.933728 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.933590 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7dc993a-dfa6-4a57-a116-87fc2a273d37","Type":"ContainerStarted","Data":"4d2aba2c1eeb766a9635e6610fb8e268ea4ebafa91d87f311849a57cdf793baa"} Jul 22 16:41:48.961124 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.960281 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-65bkg" event={"ID":"9704c18f-5f2d-417b-b49f-bf27e2774eef","Type":"ContainerStarted","Data":"d5e0041f60b0b214e73c176cd60fb12e17db019cd9588803318d5a8e4628aa48"} Jul 22 16:41:48.970392 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.970108 2557 generic.go:356] "Generic (PLEG): container finished" podID="d321f9a7-ee1f-4c09-aab2-157e849f71f3" containerID="ac30e1f1ffa4d5c00b97e46e5e22268687e2311d357b406fe01627efe75f98b0" exitCode=0 Jul 22 16:41:48.970392 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.970194 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lqf5c" event={"ID":"d321f9a7-ee1f-4c09-aab2-157e849f71f3","Type":"ContainerDied","Data":"ac30e1f1ffa4d5c00b97e46e5e22268687e2311d357b406fe01627efe75f98b0"} Jul 22 16:41:48.984944 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.983555 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc" event={"ID":"a5d6a867-9a45-4387-983a-99649541632b","Type":"ContainerStarted","Data":"27f179542bb8490364d5f120e2167eb0b9f352f98dd1ff172da7866481069998"} Jul 22 16:41:48.994878 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.993352 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-65bkg" podStartSLOduration=67.194574092 podStartE2EDuration="1m12.993331144s" totalImagesPullingTime="5.798757052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:48.984487326 +0000 UTC m=+73.961215643" watchObservedRunningTime="2026-07-22 16:41:48.993331144 +0000 UTC m=+73.970059441" Jul 22 16:41:48.994878 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.993618 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-phdn9"] Jul 22 16:41:48.995339 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:48.995258 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" event={"ID":"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e","Type":"ContainerStarted","Data":"0c6452346519ea1e609f2ad5c9a422c216a937c1e9efb3a86c18db72ce009c14"} Jul 22 16:41:49.001682 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.001612 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" event={"ID":"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db","Type":"ContainerStarted","Data":"4b6bfbf22c0839fb53a5c72103e3e9cc38e711fff4f76c883b636068e857d9fe"} Jul 22 16:41:49.006422 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:41:49.006395 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ea0909_e6e0_419a_a6cd_31bcb406be6e.slice/crio-8f6666155c584174c2c1b02f33bd9e5c05d0ccb40a76b0f7182da33bc8f3afea WatchSource:0}: Error finding container 8f6666155c584174c2c1b02f33bd9e5c05d0ccb40a76b0f7182da33bc8f3afea: Status 404 returned error can't find the container with id 8f6666155c584174c2c1b02f33bd9e5c05d0ccb40a76b0f7182da33bc8f3afea Jul 22 16:41:49.010642 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.010538 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" event={"ID":"f35a1044-4f56-4b49-b1f8-73d70c28b7b3","Type":"ContainerStarted","Data":"97e92c69d6659b4fbd6a272925e020022662bc4b6d33e95c8358892ed1a772c7"} Jul 22 16:41:49.014319 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.014294 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" event={"ID":"12249663-3e2d-425c-a1ed-76788ad82cbd","Type":"ContainerStarted","Data":"aee4768c1b0adb67e50345ef4ae282081eefdaed37812cf9d021348dcd26519b"} Jul 22 16:41:49.014435 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.014346 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" event={"ID":"12249663-3e2d-425c-a1ed-76788ad82cbd","Type":"ContainerStarted","Data":"2385ad48a7c131af829e05497e4494244b55e704dcda9e525af4dc134b279b3a"} Jul 22 16:41:49.014435 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.014375 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" event={"ID":"12249663-3e2d-425c-a1ed-76788ad82cbd","Type":"ContainerStarted","Data":"229b57ffa9d8f53704ea88aa9e46867cc5f7dcc39d7d99be3049bd84befd2c98"} Jul 22 16:41:49.100875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.100833 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:49.202699 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.202522 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:49.252641 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.252597 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 16:41:49.257342 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.257304 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.261781 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.261348 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 22 16:41:49.267099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.263969 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 22 16:41:49.267099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.264194 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 22 16:41:49.267099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.264384 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 22 16:41:49.267099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.264595 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fk2aeqp6g3ko8\"" Jul 22 16:41:49.267099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.264770 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 22 16:41:49.267099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.265006 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 22 16:41:49.267099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.265217 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 22 16:41:49.267099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.265425 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 22 16:41:49.267099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.265769 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 22 16:41:49.267099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.265976 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 22 16:41:49.267099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.266176 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 22 16:41:49.267099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.266353 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 22 16:41:49.267099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.266541 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 22 16:41:49.267099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.266730 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sptlh\"" Jul 22 16:41:49.273164 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.273142 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 22 16:41:49.291595 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.290919 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 16:41:49.307614 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.307565 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.397961 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398005 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2smf9\" (UniqueName: \"kubernetes.io/projected/634e0201-7092-4ea1-b6b4-62feecebe160-kube-api-access-2smf9\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398056 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-config\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398083 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398119 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398149 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398181 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398209 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/634e0201-7092-4ea1-b6b4-62feecebe160-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398240 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398267 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398305 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398332 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/634e0201-7092-4ea1-b6b4-62feecebe160-config-out\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398354 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398381 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398418 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-web-config\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.398875 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398442 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.399809 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398484 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.399809 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398509 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.399809 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398568 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.399809 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.398597 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.501541 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501500 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-config\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.501708 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501547 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.501708 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501581 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.501708 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501611 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.501708 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501646 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.501708 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501678 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/634e0201-7092-4ea1-b6b4-62feecebe160-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.501967 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501710 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.501967 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501739 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.501967 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501778 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.501967 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501806 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/634e0201-7092-4ea1-b6b4-62feecebe160-config-out\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.501967 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501830 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.501967 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501858 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.501967 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501898 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-web-config\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.501967 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501923 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.501967 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501963 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.502491 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.501989 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.502491 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.502019 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.502491 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.502075 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.502491 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.502128 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.502491 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.502154 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2smf9\" (UniqueName: \"kubernetes.io/projected/634e0201-7092-4ea1-b6b4-62feecebe160-kube-api-access-2smf9\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.504036 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.503998 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.507377 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.506654 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.507377 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.507070 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.507377 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.507157 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.509528 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.509500 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.511983 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.511041 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.511983 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.511221 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.511983 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.511966 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.512282 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.512184 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-config\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.512990 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.512941 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.513811 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.513753 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.514453 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.514410 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.514870 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.514806 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-web-config\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.515973 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.515508 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.515973 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.515777 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.515973 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.515930 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.516210 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.516105 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/634e0201-7092-4ea1-b6b4-62feecebe160-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.516397 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.516376 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/634e0201-7092-4ea1-b6b4-62feecebe160-config-out\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.517469 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.517441 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2smf9\" (UniqueName: \"kubernetes.io/projected/634e0201-7092-4ea1-b6b4-62feecebe160-kube-api-access-2smf9\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.518259 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.518242 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.576202 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.576168 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:41:49.619064 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.618988 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:49.678103 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:49.677940 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:50.021545 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:50.021507 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lqf5c" event={"ID":"d321f9a7-ee1f-4c09-aab2-157e849f71f3","Type":"ContainerStarted","Data":"a45071a9913b10d99838da22759d243c3df489a3e510e2f8825c7661cbbc2e44"} Jul 22 16:41:50.021744 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:50.021571 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lqf5c" event={"ID":"d321f9a7-ee1f-4c09-aab2-157e849f71f3","Type":"ContainerStarted","Data":"290a6b5f441137f6ce776966f930fde7688a572aceaf361f53ee6d49234f494e"} Jul 22 16:41:50.023232 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:50.023160 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-phdn9" event={"ID":"19ea0909-e6e0-419a-a6cd-31bcb406be6e","Type":"ContainerStarted","Data":"8f6666155c584174c2c1b02f33bd9e5c05d0ccb40a76b0f7182da33bc8f3afea"} Jul 22 16:41:50.048035 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:50.047954 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lqf5c" podStartSLOduration=6.924644038 podStartE2EDuration="8.047936588s" totalImagesPullingTime="999.520641ms" totalInitContainerRuntime="123.771909ms" isStatefulPod=true podCreationTimestamp="2026-07-22 16:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:50.044619006 +0000 UTC m=+75.021347323" watchObservedRunningTime="2026-07-22 16:41:50.047936588 +0000 UTC m=+75.024664906" Jul 22 16:41:50.132745 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:50.132691 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:50.233051 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:50.232983 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:50.350961 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:50.350884 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:50.537806 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:50.537731 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:50.642156 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:50.641929 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:51.160098 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:51.160061 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:51.262522 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:51.262486 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:53.333552 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:53.333490 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:53.391183 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:53.391133 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:53.677716 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:53.677625 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:53.778102 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:53.778065 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:54.090777 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:54.090677 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:54.192253 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:54.192208 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:54.248679 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:54.247109 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 16:41:55.048580 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.048541 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc" event={"ID":"a5d6a867-9a45-4387-983a-99649541632b","Type":"ContainerStarted","Data":"89f1b0930e55a535c0294430759a00b4d5cf40655d1af74c3d9a4957379da3ca"} Jul 22 16:41:55.049149 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.049098 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc" Jul 22 16:41:55.051577 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.051547 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" event={"ID":"ade6b05e-41d7-43fe-b7e3-0bdc444fad5e","Type":"ContainerStarted","Data":"6370571467fb5c37b98ff3d67be26dce03b37a30d230358d025e48d97997f600"} Jul 22 16:41:55.055743 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.055547 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" event={"ID":"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db","Type":"ContainerStarted","Data":"bc9d79d1af08cc35698b44ef73aa4473ec6779a9a4e540474825cdc4c2a465ab"} Jul 22 16:41:55.055743 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.055597 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" event={"ID":"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db","Type":"ContainerStarted","Data":"75cc6be06fdb9a5711bcf5a9808040393438d2e446c2952b9c4ec90012eb5a2b"} Jul 22 16:41:55.055743 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.055627 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" event={"ID":"a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db","Type":"ContainerStarted","Data":"4d10469cba7e5192ca28f2fd9019ef693d43192c65b9e137e1f8c8c015c86a1e"} Jul 22 16:41:55.055743 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.055713 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc" Jul 22 16:41:55.059334 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.059295 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d9zkt" event={"ID":"2e46da55-c83c-410b-95d5-e6d5af057a64","Type":"ContainerStarted","Data":"eb47c9bfd316a6b7889e0d7a20b354e519a0f299bca30eb51b380c749c345112"} Jul 22 16:41:55.062815 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.062792 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" event={"ID":"f35a1044-4f56-4b49-b1f8-73d70c28b7b3","Type":"ContainerStarted","Data":"5190a80f883db456d193de51193a7a845f66d8de9db9dc75658d93e5b1320371"} Jul 22 16:41:55.062916 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.062844 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" event={"ID":"f35a1044-4f56-4b49-b1f8-73d70c28b7b3","Type":"ContainerStarted","Data":"88f8a174e74bf620f4e55291c735087ee7fdfd1aa285cf3a21bb0f39fbf3a730"} Jul 22 16:41:55.062916 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.062875 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" event={"ID":"f35a1044-4f56-4b49-b1f8-73d70c28b7b3","Type":"ContainerStarted","Data":"06c545bbc36ef5f8e9099237eab24bf0a0a60d56aaeeeafb53ba98191340c908"} Jul 22 16:41:55.065472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.065448 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" event={"ID":"12249663-3e2d-425c-a1ed-76788ad82cbd","Type":"ContainerStarted","Data":"d97e476aa4970fbae7f90272629d5adca1394d6b13da8c831356013b2db51e09"} Jul 22 16:41:55.068561 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.068367 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-2ndfc" podStartSLOduration=2.92212494 podStartE2EDuration="8.068348908s" totalImagesPullingTime="5.146223968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:55.06626702 +0000 UTC m=+80.042995351" watchObservedRunningTime="2026-07-22 16:41:55.068348908 +0000 UTC m=+80.045077224" Jul 22 16:41:55.069185 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.069085 2557 generic.go:356] "Generic (PLEG): container finished" podID="634e0201-7092-4ea1-b6b4-62feecebe160" containerID="96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27" exitCode=0 Jul 22 16:41:55.069292 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.069236 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634e0201-7092-4ea1-b6b4-62feecebe160","Type":"ContainerDied","Data":"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27"} Jul 22 16:41:55.069353 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.069300 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634e0201-7092-4ea1-b6b4-62feecebe160","Type":"ContainerStarted","Data":"7f287563d6b72e47f136bb3f80ccc527e5edcc47f429cbd0f01b338d121a4b63"} Jul 22 16:41:55.072229 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.072207 2557 generic.go:356] "Generic (PLEG): container finished" podID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerID="e9e94bbb55c1093df251d193baed2dcaf5bf8674150e24353354b439d5182559" exitCode=0 Jul 22 16:41:55.072332 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.072278 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7dc993a-dfa6-4a57-a116-87fc2a273d37","Type":"ContainerDied","Data":"e9e94bbb55c1093df251d193baed2dcaf5bf8674150e24353354b439d5182559"} Jul 22 16:41:55.087695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.086620 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" podStartSLOduration=2.064306898 podStartE2EDuration="8.086600062s" totalImagesPullingTime="6.022293164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:55.085612758 +0000 UTC m=+80.062341091" watchObservedRunningTime="2026-07-22 16:41:55.086600062 +0000 UTC m=+80.063328379" Jul 22 16:41:55.106418 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.106365 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-mpf4c" podStartSLOduration=7.133330398 podStartE2EDuration="13.106350705s" totalImagesPullingTime="5.973020307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:55.105347788 +0000 UTC m=+80.082076111" watchObservedRunningTime="2026-07-22 16:41:55.106350705 +0000 UTC m=+80.083079020" Jul 22 16:41:55.144546 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.144489 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d9zkt" podStartSLOduration=2.663447317 podStartE2EDuration="22.14446899s" totalImagesPullingTime="19.481021673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 16:41:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:55.14337643 +0000 UTC m=+80.120104740" watchObservedRunningTime="2026-07-22 16:41:55.14446899 +0000 UTC m=+80.121197307" Jul 22 16:41:55.170354 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:55.170297 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-wz62t" podStartSLOduration=7.276288439 podStartE2EDuration="13.170277741s" totalImagesPullingTime="5.893989302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:55.169947077 +0000 UTC m=+80.146675418" watchObservedRunningTime="2026-07-22 16:41:55.170277741 +0000 UTC m=+80.147006058" Jul 22 16:41:56.083719 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:56.083566 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" event={"ID":"f35a1044-4f56-4b49-b1f8-73d70c28b7b3","Type":"ContainerStarted","Data":"c9b998c3b3910b5212b94f51f72adf7f7ba25caff9f5aaad69d012b2c6a5b268"} Jul 22 16:41:56.083719 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:56.083656 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" event={"ID":"f35a1044-4f56-4b49-b1f8-73d70c28b7b3","Type":"ContainerStarted","Data":"52311731c1d86a5056db4d21c2620656040f48bb4e5c48a053fe50e0df6e57d0"} Jul 22 16:41:56.083719 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:56.083705 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" event={"ID":"f35a1044-4f56-4b49-b1f8-73d70c28b7b3","Type":"ContainerStarted","Data":"40b188e9606a948287c42cd54d909ac1b9b92aff3c3bf3c23a8df11c8909c736"} Jul 22 16:41:56.084323 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:56.083753 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:41:56.108435 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:56.108374 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" podStartSLOduration=4.032522664 podStartE2EDuration="11.10835057s" totalImagesPullingTime="7.075827906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:56.106845817 +0000 UTC m=+81.083574161" watchObservedRunningTime="2026-07-22 16:41:56.10835057 +0000 UTC m=+81.085078887" Jul 22 16:41:58.098247 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:58.097828 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7dc993a-dfa6-4a57-a116-87fc2a273d37","Type":"ContainerStarted","Data":"0498d85f5280dd346dc9d65655f8a4e9adf889ef78d111ecbbecec2ad8287b30"} Jul 22 16:41:58.098247 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:58.097919 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7dc993a-dfa6-4a57-a116-87fc2a273d37","Type":"ContainerStarted","Data":"e5e30fa17ade4ebe1057ad7b317c4d8d55b15be4a605bcfbc520716ba81b913d"} Jul 22 16:41:58.098247 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:58.097961 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7dc993a-dfa6-4a57-a116-87fc2a273d37","Type":"ContainerStarted","Data":"d6296614995ec0c62f2f7bbaea92bbd40e70cbc1705942ead629a2fe5da889a9"} Jul 22 16:41:58.098247 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:58.098003 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7dc993a-dfa6-4a57-a116-87fc2a273d37","Type":"ContainerStarted","Data":"ed6d34102c14a79fcd2d658716fd01bd80649d6f13f038a57ad2e3bb2be47ce5"} Jul 22 16:41:58.098247 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:58.098061 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7dc993a-dfa6-4a57-a116-87fc2a273d37","Type":"ContainerStarted","Data":"0a3322ce561e956327f8014d0720510d57b4b562bb20dd46f7cf08c8d5ba8b17"} Jul 22 16:41:58.098247 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:58.098103 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7dc993a-dfa6-4a57-a116-87fc2a273d37","Type":"ContainerStarted","Data":"ad691577e72923cc4ed254111f0bc48d8b8a47086aad8eca6591891a76a06da4"} Jul 22 16:41:58.128456 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:58.128356 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=7.094931374 podStartE2EDuration="15.12833436s" totalImagesPullingTime="7.99360087s" totalInitContainerRuntime="39.802116ms" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:41:58.127741663 +0000 UTC m=+83.104469980" watchObservedRunningTime="2026-07-22 16:41:58.12833436 +0000 UTC m=+83.105062678" Jul 22 16:41:58.146812 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:58.146772 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:58.254410 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:58.254270 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:59.193761 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:59.193727 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:41:59.294284 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:41:59.294231 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:00.113979 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:00.113944 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634e0201-7092-4ea1-b6b4-62feecebe160","Type":"ContainerStarted","Data":"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c"} Jul 22 16:42:00.114159 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:00.114084 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634e0201-7092-4ea1-b6b4-62feecebe160","Type":"ContainerStarted","Data":"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025"} Jul 22 16:42:00.114234 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:00.114160 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634e0201-7092-4ea1-b6b4-62feecebe160","Type":"ContainerStarted","Data":"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2"} Jul 22 16:42:00.114234 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:00.114203 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634e0201-7092-4ea1-b6b4-62feecebe160","Type":"ContainerStarted","Data":"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81"} Jul 22 16:42:00.114234 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:00.114228 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634e0201-7092-4ea1-b6b4-62feecebe160","Type":"ContainerStarted","Data":"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db"} Jul 22 16:42:00.114360 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:00.114250 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634e0201-7092-4ea1-b6b4-62feecebe160","Type":"ContainerStarted","Data":"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1"} Jul 22 16:42:00.145685 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:00.145621 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.989502088 podStartE2EDuration="11.14560508s" totalImagesPullingTime="4.128979253s" totalInitContainerRuntime="27.123739ms" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:42:00.143243682 +0000 UTC m=+85.119972009" watchObservedRunningTime="2026-07-22 16:42:00.14560508 +0000 UTC m=+85.122333396" Jul 22 16:42:02.413448 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:02.413407 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:02.467660 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:02.467618 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:03.105667 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:03.105635 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7675958d48-7ndcw" Jul 22 16:42:03.345748 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:03.345703 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:03.409521 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:03.409439 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:04.576850 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:04.576811 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:42:06.742223 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:06.742186 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:07.769904 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:07.769861 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:42:07.770381 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:07.769935 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:42:08.538401 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:08.538355 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:08.593711 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:08.593668 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:09.132396 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:09.132353 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:09.233395 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:09.233356 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:10.143912 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:10.143856 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:10.164042 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:10.163980 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-phdn9" event={"ID":"19ea0909-e6e0-419a-a6cd-31bcb406be6e","Type":"ContainerStarted","Data":"b0d2cc4806989a500c64dcbf5103e7d683b9471412f5612c73d994ac512c11ad"} Jul 22 16:42:10.164383 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:10.164356 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-phdn9" Jul 22 16:42:10.183386 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:10.183363 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-phdn9" Jul 22 16:42:10.189259 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:10.189205 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-phdn9" podStartSLOduration=2.150203881 podStartE2EDuration="22.189187766s" totalImagesPullingTime="20.038983885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:42:10.188728269 +0000 UTC m=+95.165456597" watchObservedRunningTime="2026-07-22 16:42:10.189187766 +0000 UTC m=+95.165916083" Jul 22 16:42:10.214154 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:10.214120 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:13.149116 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:13.149064 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:18.249720 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:18.249684 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:18.331728 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:18.331675 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:18.432036 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:18.431987 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:18.483619 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:18.483572 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:19.017162 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:19.017131 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kqtz5" Jul 22 16:42:24.091296 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:24.091257 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:24.147813 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:24.147782 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:27.774952 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:27.774920 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:42:27.778586 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:27.778566 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-77c5f697f5-9ckm9" Jul 22 16:42:31.763313 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:31.763261 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:31.823535 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:31.823499 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:32.262400 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:32.262346 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:42:49.577203 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:49.577174 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:42:49.633787 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:49.633760 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:42:50.345981 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:42:50.345956 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:02.153592 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:02.153557 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:02.576321 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:02.576261 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:03.226942 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:03.226906 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:43:03.227339 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:03.227278 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="alertmanager" containerID="cri-o://ad691577e72923cc4ed254111f0bc48d8b8a47086aad8eca6591891a76a06da4" gracePeriod=120 Jul 22 16:43:03.227401 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:03.227332 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="kube-rbac-proxy-metric" containerID="cri-o://e5e30fa17ade4ebe1057ad7b317c4d8d55b15be4a605bcfbc520716ba81b913d" gracePeriod=120 Jul 22 16:43:03.227455 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:03.227405 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="kube-rbac-proxy-web" containerID="cri-o://ed6d34102c14a79fcd2d658716fd01bd80649d6f13f038a57ad2e3bb2be47ce5" gracePeriod=120 Jul 22 16:43:03.227455 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:03.227403 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="prom-label-proxy" containerID="cri-o://0498d85f5280dd346dc9d65655f8a4e9adf889ef78d111ecbbecec2ad8287b30" gracePeriod=120 Jul 22 16:43:03.227565 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:03.227499 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="kube-rbac-proxy" containerID="cri-o://d6296614995ec0c62f2f7bbaea92bbd40e70cbc1705942ead629a2fe5da889a9" gracePeriod=120 Jul 22 16:43:03.227565 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:03.227506 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="config-reloader" containerID="cri-o://0a3322ce561e956327f8014d0720510d57b4b562bb20dd46f7cf08c8d5ba8b17" gracePeriod=120 Jul 22 16:43:03.385512 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:03.385476 2557 generic.go:356] "Generic (PLEG): container finished" podID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerID="0498d85f5280dd346dc9d65655f8a4e9adf889ef78d111ecbbecec2ad8287b30" exitCode=0 Jul 22 16:43:03.385512 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:03.385506 2557 generic.go:356] "Generic (PLEG): container finished" podID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerID="d6296614995ec0c62f2f7bbaea92bbd40e70cbc1705942ead629a2fe5da889a9" exitCode=0 Jul 22 16:43:03.385512 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:03.385512 2557 generic.go:356] "Generic (PLEG): container finished" podID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerID="0a3322ce561e956327f8014d0720510d57b4b562bb20dd46f7cf08c8d5ba8b17" exitCode=0 Jul 22 16:43:03.385512 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:03.385517 2557 generic.go:356] "Generic (PLEG): container finished" podID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerID="ad691577e72923cc4ed254111f0bc48d8b8a47086aad8eca6591891a76a06da4" exitCode=0 Jul 22 16:43:03.385757 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:03.385545 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7dc993a-dfa6-4a57-a116-87fc2a273d37","Type":"ContainerDied","Data":"0498d85f5280dd346dc9d65655f8a4e9adf889ef78d111ecbbecec2ad8287b30"} Jul 22 16:43:03.385757 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:03.385608 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7dc993a-dfa6-4a57-a116-87fc2a273d37","Type":"ContainerDied","Data":"d6296614995ec0c62f2f7bbaea92bbd40e70cbc1705942ead629a2fe5da889a9"} Jul 22 16:43:03.385757 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:03.385635 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7dc993a-dfa6-4a57-a116-87fc2a273d37","Type":"ContainerDied","Data":"0a3322ce561e956327f8014d0720510d57b4b562bb20dd46f7cf08c8d5ba8b17"} Jul 22 16:43:03.385757 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:03.385662 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7dc993a-dfa6-4a57-a116-87fc2a273d37","Type":"ContainerDied","Data":"ad691577e72923cc4ed254111f0bc48d8b8a47086aad8eca6591891a76a06da4"} Jul 22 16:43:04.394192 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.394162 2557 generic.go:356] "Generic (PLEG): container finished" podID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerID="e5e30fa17ade4ebe1057ad7b317c4d8d55b15be4a605bcfbc520716ba81b913d" exitCode=0 Jul 22 16:43:04.394527 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.394195 2557 generic.go:356] "Generic (PLEG): container finished" podID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerID="ed6d34102c14a79fcd2d658716fd01bd80649d6f13f038a57ad2e3bb2be47ce5" exitCode=0 Jul 22 16:43:04.394527 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.394227 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7dc993a-dfa6-4a57-a116-87fc2a273d37","Type":"ContainerDied","Data":"e5e30fa17ade4ebe1057ad7b317c4d8d55b15be4a605bcfbc520716ba81b913d"} Jul 22 16:43:04.394527 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.394292 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7dc993a-dfa6-4a57-a116-87fc2a273d37","Type":"ContainerDied","Data":"ed6d34102c14a79fcd2d658716fd01bd80649d6f13f038a57ad2e3bb2be47ce5"} Jul 22 16:43:04.468454 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.468428 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:04.583924 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.583844 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7dc993a-dfa6-4a57-a116-87fc2a273d37-tls-assets\") pod \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " Jul 22 16:43:04.583924 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.583896 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7dc993a-dfa6-4a57-a116-87fc2a273d37-alertmanager-trusted-ca-bundle\") pod \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " Jul 22 16:43:04.584168 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.583936 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzpvs\" (UniqueName: \"kubernetes.io/projected/d7dc993a-dfa6-4a57-a116-87fc2a273d37-kube-api-access-dzpvs\") pod \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " Jul 22 16:43:04.584168 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.583964 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-config-volume\") pod \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " Jul 22 16:43:04.584168 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.583988 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-web-config\") pod \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " Jul 22 16:43:04.584168 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.584081 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy\") pod \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " Jul 22 16:43:04.584168 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.584109 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7dc993a-dfa6-4a57-a116-87fc2a273d37-alertmanager-main-db\") pod \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " Jul 22 16:43:04.584168 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.584137 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-cluster-tls-config\") pod \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " Jul 22 16:43:04.584168 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.584164 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-main-tls\") pod \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " Jul 22 16:43:04.584493 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.584189 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy-web\") pod \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " Jul 22 16:43:04.584493 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.584217 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7dc993a-dfa6-4a57-a116-87fc2a273d37-metrics-client-ca\") pod \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " Jul 22 16:43:04.584493 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.584259 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " Jul 22 16:43:04.584493 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.584290 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7dc993a-dfa6-4a57-a116-87fc2a273d37-config-out\") pod \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\" (UID: \"d7dc993a-dfa6-4a57-a116-87fc2a273d37\") " Jul 22 16:43:04.584493 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.584378 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7dc993a-dfa6-4a57-a116-87fc2a273d37-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d7dc993a-dfa6-4a57-a116-87fc2a273d37" (UID: "d7dc993a-dfa6-4a57-a116-87fc2a273d37"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:43:04.584725 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.584580 2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7dc993a-dfa6-4a57-a116-87fc2a273d37-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:04.584725 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.584705 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7dc993a-dfa6-4a57-a116-87fc2a273d37-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d7dc993a-dfa6-4a57-a116-87fc2a273d37" (UID: "d7dc993a-dfa6-4a57-a116-87fc2a273d37"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:43:04.585145 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.585116 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7dc993a-dfa6-4a57-a116-87fc2a273d37-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d7dc993a-dfa6-4a57-a116-87fc2a273d37" (UID: "d7dc993a-dfa6-4a57-a116-87fc2a273d37"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:43:04.587738 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.587482 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d7dc993a-dfa6-4a57-a116-87fc2a273d37" (UID: "d7dc993a-dfa6-4a57-a116-87fc2a273d37"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:04.587874 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.587824 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-config-volume" (OuterVolumeSpecName: "config-volume") pod "d7dc993a-dfa6-4a57-a116-87fc2a273d37" (UID: "d7dc993a-dfa6-4a57-a116-87fc2a273d37"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:04.588365 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.588257 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d7dc993a-dfa6-4a57-a116-87fc2a273d37" (UID: "d7dc993a-dfa6-4a57-a116-87fc2a273d37"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:04.588365 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.588261 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7dc993a-dfa6-4a57-a116-87fc2a273d37-kube-api-access-dzpvs" (OuterVolumeSpecName: "kube-api-access-dzpvs") pod "d7dc993a-dfa6-4a57-a116-87fc2a273d37" (UID: "d7dc993a-dfa6-4a57-a116-87fc2a273d37"). InnerVolumeSpecName "kube-api-access-dzpvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:43:04.588595 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.588463 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d7dc993a-dfa6-4a57-a116-87fc2a273d37" (UID: "d7dc993a-dfa6-4a57-a116-87fc2a273d37"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:04.588595 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.588558 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7dc993a-dfa6-4a57-a116-87fc2a273d37-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d7dc993a-dfa6-4a57-a116-87fc2a273d37" (UID: "d7dc993a-dfa6-4a57-a116-87fc2a273d37"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:43:04.589791 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.589771 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d7dc993a-dfa6-4a57-a116-87fc2a273d37" (UID: "d7dc993a-dfa6-4a57-a116-87fc2a273d37"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:04.589965 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.589948 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7dc993a-dfa6-4a57-a116-87fc2a273d37-config-out" (OuterVolumeSpecName: "config-out") pod "d7dc993a-dfa6-4a57-a116-87fc2a273d37" (UID: "d7dc993a-dfa6-4a57-a116-87fc2a273d37"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:43:04.592697 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.592639 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d7dc993a-dfa6-4a57-a116-87fc2a273d37" (UID: "d7dc993a-dfa6-4a57-a116-87fc2a273d37"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:04.599118 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.599093 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-web-config" (OuterVolumeSpecName: "web-config") pod "d7dc993a-dfa6-4a57-a116-87fc2a273d37" (UID: "d7dc993a-dfa6-4a57-a116-87fc2a273d37"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:04.685765 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.685731 2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:04.685765 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.685760 2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d7dc993a-dfa6-4a57-a116-87fc2a273d37-alertmanager-main-db\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:04.685765 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.685770 2557 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-cluster-tls-config\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:04.685948 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.685779 2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-main-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:04.685948 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.685788 2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:04.685948 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.685797 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7dc993a-dfa6-4a57-a116-87fc2a273d37-metrics-client-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:04.685948 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.685805 2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:04.685948 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.685814 2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7dc993a-dfa6-4a57-a116-87fc2a273d37-config-out\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:04.685948 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.685823 2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7dc993a-dfa6-4a57-a116-87fc2a273d37-tls-assets\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:04.685948 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.685830 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzpvs\" (UniqueName: \"kubernetes.io/projected/d7dc993a-dfa6-4a57-a116-87fc2a273d37-kube-api-access-dzpvs\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:04.685948 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.685838 2557 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-config-volume\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:04.685948 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.685846 2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7dc993a-dfa6-4a57-a116-87fc2a273d37-web-config\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:04.853779 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:04.853696 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:05.401433 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.401405 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.401839 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.401405 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d7dc993a-dfa6-4a57-a116-87fc2a273d37","Type":"ContainerDied","Data":"4d2aba2c1eeb766a9635e6610fb8e268ea4ebafa91d87f311849a57cdf793baa"} Jul 22 16:43:05.401839 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.401564 2557 scope.go:117] "RemoveContainer" containerID="0498d85f5280dd346dc9d65655f8a4e9adf889ef78d111ecbbecec2ad8287b30" Jul 22 16:43:05.410328 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.410306 2557 scope.go:117] "RemoveContainer" containerID="e5e30fa17ade4ebe1057ad7b317c4d8d55b15be4a605bcfbc520716ba81b913d" Jul 22 16:43:05.417382 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.417362 2557 scope.go:117] "RemoveContainer" containerID="d6296614995ec0c62f2f7bbaea92bbd40e70cbc1705942ead629a2fe5da889a9" Jul 22 16:43:05.424270 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.424227 2557 scope.go:117] "RemoveContainer" containerID="ed6d34102c14a79fcd2d658716fd01bd80649d6f13f038a57ad2e3bb2be47ce5" Jul 22 16:43:05.426602 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.426547 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:43:05.430049 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.430012 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:43:05.432422 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.432403 2557 scope.go:117] "RemoveContainer" containerID="0a3322ce561e956327f8014d0720510d57b4b562bb20dd46f7cf08c8d5ba8b17" Jul 22 16:43:05.439506 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.439485 2557 scope.go:117] "RemoveContainer" containerID="ad691577e72923cc4ed254111f0bc48d8b8a47086aad8eca6591891a76a06da4" Jul 22 16:43:05.446508 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.446373 2557 scope.go:117] "RemoveContainer" containerID="e9e94bbb55c1093df251d193baed2dcaf5bf8674150e24353354b439d5182559" Jul 22 16:43:05.458600 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.458576 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:43:05.459035 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459005 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="prom-label-proxy" Jul 22 16:43:05.459125 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459047 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="prom-label-proxy" Jul 22 16:43:05.459125 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459067 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="init-config-reloader" Jul 22 16:43:05.459125 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459077 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="init-config-reloader" Jul 22 16:43:05.459125 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459088 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="config-reloader" Jul 22 16:43:05.459125 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459096 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="config-reloader" Jul 22 16:43:05.459125 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459107 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="alertmanager" Jul 22 16:43:05.459125 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459115 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="alertmanager" Jul 22 16:43:05.459472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459127 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="kube-rbac-proxy-web" Jul 22 16:43:05.459472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459137 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="kube-rbac-proxy-web" Jul 22 16:43:05.459472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459152 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="kube-rbac-proxy" Jul 22 16:43:05.459472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459159 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="kube-rbac-proxy" Jul 22 16:43:05.459472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459183 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="kube-rbac-proxy-metric" Jul 22 16:43:05.459472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459192 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="kube-rbac-proxy-metric" Jul 22 16:43:05.459472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459279 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="prom-label-proxy" Jul 22 16:43:05.459472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459294 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="alertmanager" Jul 22 16:43:05.459472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459306 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="kube-rbac-proxy" Jul 22 16:43:05.459472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459315 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="config-reloader" Jul 22 16:43:05.459472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459326 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="kube-rbac-proxy-web" Jul 22 16:43:05.459472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.459337 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" containerName="kube-rbac-proxy-metric" Jul 22 16:43:05.463058 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.463016 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.465821 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.465801 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 16:43:05.465821 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.465802 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 16:43:05.466133 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.466079 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 16:43:05.466133 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.466083 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 16:43:05.466133 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.466091 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 16:43:05.466357 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.466134 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 16:43:05.466357 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.466147 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 16:43:05.466357 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.466153 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-84ksk\"" Jul 22 16:43:05.466532 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.466518 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 16:43:05.473739 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.473713 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 16:43:05.476092 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.475990 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:43:05.573672 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.573592 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7dc993a-dfa6-4a57-a116-87fc2a273d37" path="/var/lib/kubelet/pods/d7dc993a-dfa6-4a57-a116-87fc2a273d37/volumes" Jul 22 16:43:05.592934 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.592898 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b4e4af1-8bab-400e-bdf0-d030a12220da-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.593088 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.592947 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smfdv\" (UniqueName: \"kubernetes.io/projected/5b4e4af1-8bab-400e-bdf0-d030a12220da-kube-api-access-smfdv\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.593088 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.592988 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-config-volume\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.593088 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.593016 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-web-config\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.593088 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.593055 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5b4e4af1-8bab-400e-bdf0-d030a12220da-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.593235 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.593117 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.593235 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.593150 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b4e4af1-8bab-400e-bdf0-d030a12220da-config-out\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.593235 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.593187 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.593235 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.593211 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b4e4af1-8bab-400e-bdf0-d030a12220da-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.593351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.593243 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.593351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.593268 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.593351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.593284 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b4e4af1-8bab-400e-bdf0-d030a12220da-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.593351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.593306 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.694240 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.694201 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.694240 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.694243 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b4e4af1-8bab-400e-bdf0-d030a12220da-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.694472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.694263 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.694472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.694281 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.694472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.694305 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b4e4af1-8bab-400e-bdf0-d030a12220da-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.694472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.694335 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.694472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.694373 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b4e4af1-8bab-400e-bdf0-d030a12220da-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.694704 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.694531 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smfdv\" (UniqueName: \"kubernetes.io/projected/5b4e4af1-8bab-400e-bdf0-d030a12220da-kube-api-access-smfdv\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.694704 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.694605 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-config-volume\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.694704 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.694632 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-web-config\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.694704 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.694660 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5b4e4af1-8bab-400e-bdf0-d030a12220da-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.694704 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.694698 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.694946 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.694737 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b4e4af1-8bab-400e-bdf0-d030a12220da-config-out\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.695397 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.695120 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b4e4af1-8bab-400e-bdf0-d030a12220da-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.695397 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.695320 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b4e4af1-8bab-400e-bdf0-d030a12220da-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.695936 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.695912 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5b4e4af1-8bab-400e-bdf0-d030a12220da-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.697416 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.697266 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b4e4af1-8bab-400e-bdf0-d030a12220da-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.697416 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.697382 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.697578 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.697501 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b4e4af1-8bab-400e-bdf0-d030a12220da-config-out\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.697708 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.697687 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-web-config\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.698052 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.698007 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.698115 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.698010 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.698427 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.698404 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.698634 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.698618 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.699414 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.699394 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5b4e4af1-8bab-400e-bdf0-d030a12220da-config-volume\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.703511 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.703491 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smfdv\" (UniqueName: \"kubernetes.io/projected/5b4e4af1-8bab-400e-bdf0-d030a12220da-kube-api-access-smfdv\") pod \"alertmanager-main-0\" (UID: \"5b4e4af1-8bab-400e-bdf0-d030a12220da\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.773174 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.773142 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 16:43:05.910646 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:05.910622 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 16:43:05.912697 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:43:05.912672 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4e4af1_8bab_400e_bdf0_d030a12220da.slice/crio-e99f46885442381bf84003d40a5f5197f1b1a146387b961967bffb3be7306ed7 WatchSource:0}: Error finding container e99f46885442381bf84003d40a5f5197f1b1a146387b961967bffb3be7306ed7: Status 404 returned error can't find the container with id e99f46885442381bf84003d40a5f5197f1b1a146387b961967bffb3be7306ed7 Jul 22 16:43:06.406673 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:06.406636 2557 generic.go:356] "Generic (PLEG): container finished" podID="5b4e4af1-8bab-400e-bdf0-d030a12220da" containerID="08ce062f1fbed65fc2944146bbc56b87dd92d9b6d07d68a45b3d5390b0a0db6a" exitCode=0 Jul 22 16:43:06.407158 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:06.406724 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b4e4af1-8bab-400e-bdf0-d030a12220da","Type":"ContainerDied","Data":"08ce062f1fbed65fc2944146bbc56b87dd92d9b6d07d68a45b3d5390b0a0db6a"} Jul 22 16:43:06.407158 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:06.406796 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b4e4af1-8bab-400e-bdf0-d030a12220da","Type":"ContainerStarted","Data":"e99f46885442381bf84003d40a5f5197f1b1a146387b961967bffb3be7306ed7"} Jul 22 16:43:07.416370 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.416337 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b4e4af1-8bab-400e-bdf0-d030a12220da","Type":"ContainerStarted","Data":"d458be185fa1e14e1663ad0242e6a78464a35763c6a12d2e0220d841d7935b4d"} Jul 22 16:43:07.416781 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.416400 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b4e4af1-8bab-400e-bdf0-d030a12220da","Type":"ContainerStarted","Data":"fb7fa73bf48e4c8cea048c4f1c0828260f15817cd75e96c28bbc17fc184cf077"} Jul 22 16:43:07.416781 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.416425 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b4e4af1-8bab-400e-bdf0-d030a12220da","Type":"ContainerStarted","Data":"5a8a86be81bca0cbd17445ecb0ecb5b8572c06f18a75acc82b1374f86149e92f"} Jul 22 16:43:07.416781 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.416459 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b4e4af1-8bab-400e-bdf0-d030a12220da","Type":"ContainerStarted","Data":"5908dc942fa63b11e806d0b727eac82608f303e055cedcdfea4127677bd5bf32"} Jul 22 16:43:07.416781 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.416489 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b4e4af1-8bab-400e-bdf0-d030a12220da","Type":"ContainerStarted","Data":"2458ce2c8c37063344ba4cc7930cce5f1c69bbe6b70ffb70e7b50bb68902dbcc"} Jul 22 16:43:07.416781 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.416518 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5b4e4af1-8bab-400e-bdf0-d030a12220da","Type":"ContainerStarted","Data":"ed18bf6e83ca4cca8f045b4ce744d78666b3c9931b370bec26d5d29c5b5c9749"} Jul 22 16:43:07.444351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.444290 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.428420596 podStartE2EDuration="2.444266762s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.846166ms" isStatefulPod=false podCreationTimestamp="2026-07-22 16:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:43:07.441636306 +0000 UTC m=+152.418364658" watchObservedRunningTime="2026-07-22 16:43:07.444266762 +0000 UTC m=+152.420995079" Jul 22 16:43:07.535060 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.534983 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 16:43:07.535565 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.535362 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="prometheus" containerID="cri-o://7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1" gracePeriod=600 Jul 22 16:43:07.535565 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.535391 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c" gracePeriod=600 Jul 22 16:43:07.535565 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.535411 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="kube-rbac-proxy-web" containerID="cri-o://8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2" gracePeriod=600 Jul 22 16:43:07.535565 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.535460 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="config-reloader" containerID="cri-o://b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db" gracePeriod=600 Jul 22 16:43:07.535565 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.535362 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="kube-rbac-proxy" containerID="cri-o://b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025" gracePeriod=600 Jul 22 16:43:07.535565 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.535388 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="thanos-sidecar" containerID="cri-o://452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81" gracePeriod=600 Jul 22 16:43:07.790132 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.790062 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:07.915108 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915076 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-thanos-prometheus-http-client-file\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915300 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915120 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-1\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915300 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915143 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/634e0201-7092-4ea1-b6b4-62feecebe160-tls-assets\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915300 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915167 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915300 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915184 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/634e0201-7092-4ea1-b6b4-62feecebe160-config-out\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915518 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915316 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915518 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915382 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-db\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915518 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915466 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:43:07.915672 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915520 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-web-config\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915672 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915559 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-metrics-client-certs\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915672 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915593 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-kubelet-serving-ca-bundle\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915672 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915626 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-config\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915672 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915657 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-serving-certs-ca-bundle\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915908 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915682 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-2\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915908 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915721 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-0\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915908 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915751 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2smf9\" (UniqueName: \"kubernetes.io/projected/634e0201-7092-4ea1-b6b4-62feecebe160-kube-api-access-2smf9\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915908 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915787 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-metrics-client-ca\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915908 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915819 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-grpc-tls\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915908 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915855 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-kube-rbac-proxy\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.915908 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915885 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-tls\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.916240 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.915918 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-trusted-ca-bundle\") pod \"634e0201-7092-4ea1-b6b4-62feecebe160\" (UID: \"634e0201-7092-4ea1-b6b4-62feecebe160\") " Jul 22 16:43:07.916240 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.916227 2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:07.916350 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.916322 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:43:07.916529 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.916508 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:43:07.916601 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.916582 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:43:07.916928 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.916900 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:43:07.917998 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.917849 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634e0201-7092-4ea1-b6b4-62feecebe160-config-out" (OuterVolumeSpecName: "config-out") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:43:07.918643 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.918251 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634e0201-7092-4ea1-b6b4-62feecebe160-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:43:07.918643 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.918318 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:07.918643 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.918410 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:07.918643 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.918448 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:43:07.918643 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.918507 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:43:07.918643 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.918608 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:07.918977 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.918672 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:43:07.918977 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.918908 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:07.920201 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.920177 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634e0201-7092-4ea1-b6b4-62feecebe160-kube-api-access-2smf9" (OuterVolumeSpecName: "kube-api-access-2smf9") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "kube-api-access-2smf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:43:07.920407 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.920388 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:07.920568 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.920539 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:07.920814 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.920799 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:07.920880 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.920848 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-config" (OuterVolumeSpecName: "config") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:07.929511 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:07.929491 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-web-config" (OuterVolumeSpecName: "web-config") pod "634e0201-7092-4ea1-b6b4-62feecebe160" (UID: "634e0201-7092-4ea1-b6b4-62feecebe160"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:43:08.017531 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017505 2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/634e0201-7092-4ea1-b6b4-62feecebe160-tls-assets\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017531 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017530 2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017539 2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/634e0201-7092-4ea1-b6b4-62feecebe160-config-out\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017550 2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017559 2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-db\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017567 2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-web-config\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017576 2557 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-metrics-client-certs\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017585 2557 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017593 2557 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-config\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017602 2557 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017610 2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017620 2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017630 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2smf9\" (UniqueName: \"kubernetes.io/projected/634e0201-7092-4ea1-b6b4-62feecebe160-kube-api-access-2smf9\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017638 2557 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-configmap-metrics-client-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017647 2557 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-grpc-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017655 2557 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-kube-rbac-proxy\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017665 2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017673 2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634e0201-7092-4ea1-b6b4-62feecebe160-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.017695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.017682 2557 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/634e0201-7092-4ea1-b6b4-62feecebe160-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:43:08.424716 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.424672 2557 generic.go:356] "Generic (PLEG): container finished" podID="634e0201-7092-4ea1-b6b4-62feecebe160" containerID="ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c" exitCode=0 Jul 22 16:43:08.424716 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.424698 2557 generic.go:356] "Generic (PLEG): container finished" podID="634e0201-7092-4ea1-b6b4-62feecebe160" containerID="b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025" exitCode=0 Jul 22 16:43:08.424716 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.424705 2557 generic.go:356] "Generic (PLEG): container finished" podID="634e0201-7092-4ea1-b6b4-62feecebe160" containerID="8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2" exitCode=0 Jul 22 16:43:08.424716 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.424711 2557 generic.go:356] "Generic (PLEG): container finished" podID="634e0201-7092-4ea1-b6b4-62feecebe160" containerID="452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81" exitCode=0 Jul 22 16:43:08.424716 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.424716 2557 generic.go:356] "Generic (PLEG): container finished" podID="634e0201-7092-4ea1-b6b4-62feecebe160" containerID="b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db" exitCode=0 Jul 22 16:43:08.424716 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.424721 2557 generic.go:356] "Generic (PLEG): container finished" podID="634e0201-7092-4ea1-b6b4-62feecebe160" containerID="7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1" exitCode=0 Jul 22 16:43:08.425470 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.424748 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.425470 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.424752 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634e0201-7092-4ea1-b6b4-62feecebe160","Type":"ContainerDied","Data":"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c"} Jul 22 16:43:08.425470 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.424840 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634e0201-7092-4ea1-b6b4-62feecebe160","Type":"ContainerDied","Data":"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025"} Jul 22 16:43:08.425470 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.424891 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634e0201-7092-4ea1-b6b4-62feecebe160","Type":"ContainerDied","Data":"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2"} Jul 22 16:43:08.425470 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.424904 2557 scope.go:117] "RemoveContainer" containerID="ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c" Jul 22 16:43:08.425470 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.424929 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634e0201-7092-4ea1-b6b4-62feecebe160","Type":"ContainerDied","Data":"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81"} Jul 22 16:43:08.425470 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.424972 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634e0201-7092-4ea1-b6b4-62feecebe160","Type":"ContainerDied","Data":"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db"} Jul 22 16:43:08.425470 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.425012 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634e0201-7092-4ea1-b6b4-62feecebe160","Type":"ContainerDied","Data":"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1"} Jul 22 16:43:08.425470 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.425071 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634e0201-7092-4ea1-b6b4-62feecebe160","Type":"ContainerDied","Data":"7f287563d6b72e47f136bb3f80ccc527e5edcc47f429cbd0f01b338d121a4b63"} Jul 22 16:43:08.432944 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.432923 2557 scope.go:117] "RemoveContainer" containerID="b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025" Jul 22 16:43:08.439909 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.439885 2557 scope.go:117] "RemoveContainer" containerID="8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2" Jul 22 16:43:08.446917 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.446858 2557 scope.go:117] "RemoveContainer" containerID="452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81" Jul 22 16:43:08.449641 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.449618 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 16:43:08.455070 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.455005 2557 scope.go:117] "RemoveContainer" containerID="b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db" Jul 22 16:43:08.455070 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.455053 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 16:43:08.461761 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.461742 2557 scope.go:117] "RemoveContainer" containerID="7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1" Jul 22 16:43:08.469047 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.469012 2557 scope.go:117] "RemoveContainer" containerID="96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27" Jul 22 16:43:08.475799 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.475783 2557 scope.go:117] "RemoveContainer" containerID="ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c" Jul 22 16:43:08.476079 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:43:08.476061 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c\": container with ID starting with ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c not found: ID does not exist" containerID="ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c" Jul 22 16:43:08.476124 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.476087 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c"} err="failed to get container status \"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c\": rpc error: code = NotFound desc = could not find container \"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c\": container with ID starting with ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c not found: ID does not exist" Jul 22 16:43:08.476124 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.476120 2557 scope.go:117] "RemoveContainer" containerID="b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025" Jul 22 16:43:08.476342 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:43:08.476319 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025\": container with ID starting with b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025 not found: ID does not exist" containerID="b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025" Jul 22 16:43:08.476380 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.476348 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025"} err="failed to get container status \"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025\": rpc error: code = NotFound desc = could not find container \"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025\": container with ID starting with b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025 not found: ID does not exist" Jul 22 16:43:08.476380 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.476365 2557 scope.go:117] "RemoveContainer" containerID="8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2" Jul 22 16:43:08.476575 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:43:08.476558 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2\": container with ID starting with 8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2 not found: ID does not exist" containerID="8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2" Jul 22 16:43:08.476612 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.476579 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2"} err="failed to get container status \"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2\": rpc error: code = NotFound desc = could not find container \"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2\": container with ID starting with 8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2 not found: ID does not exist" Jul 22 16:43:08.476612 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.476592 2557 scope.go:117] "RemoveContainer" containerID="452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81" Jul 22 16:43:08.476782 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:43:08.476768 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81\": container with ID starting with 452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81 not found: ID does not exist" containerID="452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81" Jul 22 16:43:08.476824 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.476785 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81"} err="failed to get container status \"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81\": rpc error: code = NotFound desc = could not find container \"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81\": container with ID starting with 452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81 not found: ID does not exist" Jul 22 16:43:08.476824 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.476800 2557 scope.go:117] "RemoveContainer" containerID="b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db" Jul 22 16:43:08.477043 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:43:08.477005 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db\": container with ID starting with b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db not found: ID does not exist" containerID="b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db" Jul 22 16:43:08.477093 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.477045 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db"} err="failed to get container status \"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db\": rpc error: code = NotFound desc = could not find container \"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db\": container with ID starting with b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db not found: ID does not exist" Jul 22 16:43:08.477093 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.477058 2557 scope.go:117] "RemoveContainer" containerID="7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1" Jul 22 16:43:08.477246 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:43:08.477231 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1\": container with ID starting with 7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1 not found: ID does not exist" containerID="7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1" Jul 22 16:43:08.477296 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.477248 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1"} err="failed to get container status \"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1\": rpc error: code = NotFound desc = could not find container \"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1\": container with ID starting with 7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1 not found: ID does not exist" Jul 22 16:43:08.477296 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.477259 2557 scope.go:117] "RemoveContainer" containerID="96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27" Jul 22 16:43:08.477449 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:43:08.477435 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27\": container with ID starting with 96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27 not found: ID does not exist" containerID="96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27" Jul 22 16:43:08.477488 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.477450 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27"} err="failed to get container status \"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27\": rpc error: code = NotFound desc = could not find container \"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27\": container with ID starting with 96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27 not found: ID does not exist" Jul 22 16:43:08.477488 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.477460 2557 scope.go:117] "RemoveContainer" containerID="ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c" Jul 22 16:43:08.477642 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.477627 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c"} err="failed to get container status \"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c\": rpc error: code = NotFound desc = could not find container \"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c\": container with ID starting with ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c not found: ID does not exist" Jul 22 16:43:08.477680 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.477642 2557 scope.go:117] "RemoveContainer" containerID="b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025" Jul 22 16:43:08.477824 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.477810 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025"} err="failed to get container status \"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025\": rpc error: code = NotFound desc = could not find container \"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025\": container with ID starting with b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025 not found: ID does not exist" Jul 22 16:43:08.477865 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.477823 2557 scope.go:117] "RemoveContainer" containerID="8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2" Jul 22 16:43:08.478041 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.478005 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2"} err="failed to get container status \"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2\": rpc error: code = NotFound desc = could not find container \"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2\": container with ID starting with 8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2 not found: ID does not exist" Jul 22 16:43:08.478092 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.478042 2557 scope.go:117] "RemoveContainer" containerID="452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81" Jul 22 16:43:08.478252 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.478232 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81"} err="failed to get container status \"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81\": rpc error: code = NotFound desc = could not find container \"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81\": container with ID starting with 452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81 not found: ID does not exist" Jul 22 16:43:08.478298 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.478254 2557 scope.go:117] "RemoveContainer" containerID="b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db" Jul 22 16:43:08.478446 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.478430 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db"} err="failed to get container status \"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db\": rpc error: code = NotFound desc = could not find container \"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db\": container with ID starting with b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db not found: ID does not exist" Jul 22 16:43:08.478446 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.478444 2557 scope.go:117] "RemoveContainer" containerID="7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1" Jul 22 16:43:08.478676 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.478660 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1"} err="failed to get container status \"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1\": rpc error: code = NotFound desc = could not find container \"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1\": container with ID starting with 7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1 not found: ID does not exist" Jul 22 16:43:08.478718 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.478677 2557 scope.go:117] "RemoveContainer" containerID="96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27" Jul 22 16:43:08.478874 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.478858 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27"} err="failed to get container status \"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27\": rpc error: code = NotFound desc = could not find container \"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27\": container with ID starting with 96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27 not found: ID does not exist" Jul 22 16:43:08.478919 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.478874 2557 scope.go:117] "RemoveContainer" containerID="ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c" Jul 22 16:43:08.479105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.479086 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c"} err="failed to get container status \"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c\": rpc error: code = NotFound desc = could not find container \"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c\": container with ID starting with ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c not found: ID does not exist" Jul 22 16:43:08.479153 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.479107 2557 scope.go:117] "RemoveContainer" containerID="b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025" Jul 22 16:43:08.479316 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.479297 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025"} err="failed to get container status \"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025\": rpc error: code = NotFound desc = could not find container \"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025\": container with ID starting with b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025 not found: ID does not exist" Jul 22 16:43:08.479392 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.479320 2557 scope.go:117] "RemoveContainer" containerID="8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2" Jul 22 16:43:08.479538 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.479518 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2"} err="failed to get container status \"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2\": rpc error: code = NotFound desc = could not find container \"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2\": container with ID starting with 8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2 not found: ID does not exist" Jul 22 16:43:08.479586 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.479539 2557 scope.go:117] "RemoveContainer" containerID="452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81" Jul 22 16:43:08.479744 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.479726 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81"} err="failed to get container status \"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81\": rpc error: code = NotFound desc = could not find container \"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81\": container with ID starting with 452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81 not found: ID does not exist" Jul 22 16:43:08.479822 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.479744 2557 scope.go:117] "RemoveContainer" containerID="b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db" Jul 22 16:43:08.479949 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.479931 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db"} err="failed to get container status \"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db\": rpc error: code = NotFound desc = could not find container \"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db\": container with ID starting with b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db not found: ID does not exist" Jul 22 16:43:08.480006 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.479949 2557 scope.go:117] "RemoveContainer" containerID="7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1" Jul 22 16:43:08.480424 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.480275 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1"} err="failed to get container status \"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1\": rpc error: code = NotFound desc = could not find container \"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1\": container with ID starting with 7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1 not found: ID does not exist" Jul 22 16:43:08.480424 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.480294 2557 scope.go:117] "RemoveContainer" containerID="96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27" Jul 22 16:43:08.480681 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.480657 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27"} err="failed to get container status \"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27\": rpc error: code = NotFound desc = could not find container \"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27\": container with ID starting with 96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27 not found: ID does not exist" Jul 22 16:43:08.480814 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.480683 2557 scope.go:117] "RemoveContainer" containerID="ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c" Jul 22 16:43:08.481138 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.481106 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c"} err="failed to get container status \"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c\": rpc error: code = NotFound desc = could not find container \"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c\": container with ID starting with ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c not found: ID does not exist" Jul 22 16:43:08.481225 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.481152 2557 scope.go:117] "RemoveContainer" containerID="b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025" Jul 22 16:43:08.481429 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.481409 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025"} err="failed to get container status \"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025\": rpc error: code = NotFound desc = could not find container \"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025\": container with ID starting with b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025 not found: ID does not exist" Jul 22 16:43:08.481429 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.481428 2557 scope.go:117] "RemoveContainer" containerID="8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2" Jul 22 16:43:08.481690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.481642 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2"} err="failed to get container status \"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2\": rpc error: code = NotFound desc = could not find container \"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2\": container with ID starting with 8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2 not found: ID does not exist" Jul 22 16:43:08.481690 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.481663 2557 scope.go:117] "RemoveContainer" containerID="452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81" Jul 22 16:43:08.481872 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.481854 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81"} err="failed to get container status \"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81\": rpc error: code = NotFound desc = could not find container \"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81\": container with ID starting with 452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81 not found: ID does not exist" Jul 22 16:43:08.481934 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.481873 2557 scope.go:117] "RemoveContainer" containerID="b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db" Jul 22 16:43:08.482139 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482116 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db"} err="failed to get container status \"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db\": rpc error: code = NotFound desc = could not find container \"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db\": container with ID starting with b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db not found: ID does not exist" Jul 22 16:43:08.482201 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482140 2557 scope.go:117] "RemoveContainer" containerID="7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1" Jul 22 16:43:08.482335 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482319 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1"} err="failed to get container status \"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1\": rpc error: code = NotFound desc = could not find container \"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1\": container with ID starting with 7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1 not found: ID does not exist" Jul 22 16:43:08.482377 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482335 2557 scope.go:117] "RemoveContainer" containerID="96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27" Jul 22 16:43:08.482501 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482483 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 16:43:08.482558 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482542 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27"} err="failed to get container status \"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27\": rpc error: code = NotFound desc = could not find container \"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27\": container with ID starting with 96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27 not found: ID does not exist" Jul 22 16:43:08.482608 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482559 2557 scope.go:117] "RemoveContainer" containerID="ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c" Jul 22 16:43:08.482748 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482729 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c"} err="failed to get container status \"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c\": rpc error: code = NotFound desc = could not find container \"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c\": container with ID starting with ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c not found: ID does not exist" Jul 22 16:43:08.482823 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482749 2557 scope.go:117] "RemoveContainer" containerID="b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025" Jul 22 16:43:08.482864 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482836 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="kube-rbac-proxy" Jul 22 16:43:08.482864 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482855 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="kube-rbac-proxy" Jul 22 16:43:08.482961 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482874 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="thanos-sidecar" Jul 22 16:43:08.482961 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482883 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="thanos-sidecar" Jul 22 16:43:08.482961 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482902 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="kube-rbac-proxy-web" Jul 22 16:43:08.482961 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482911 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="kube-rbac-proxy-web" Jul 22 16:43:08.482961 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482925 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="prometheus" Jul 22 16:43:08.482961 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482931 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="prometheus" Jul 22 16:43:08.482961 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482940 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="init-config-reloader" Jul 22 16:43:08.482961 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482945 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="init-config-reloader" Jul 22 16:43:08.482961 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482952 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="kube-rbac-proxy-thanos" Jul 22 16:43:08.482961 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482957 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="kube-rbac-proxy-thanos" Jul 22 16:43:08.482961 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482965 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="config-reloader" Jul 22 16:43:08.483351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482971 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="config-reloader" Jul 22 16:43:08.483351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482960 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025"} err="failed to get container status \"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025\": rpc error: code = NotFound desc = could not find container \"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025\": container with ID starting with b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025 not found: ID does not exist" Jul 22 16:43:08.483351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.482983 2557 scope.go:117] "RemoveContainer" containerID="8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2" Jul 22 16:43:08.483351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.483055 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="kube-rbac-proxy" Jul 22 16:43:08.483351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.483065 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="prometheus" Jul 22 16:43:08.483351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.483071 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="kube-rbac-proxy-thanos" Jul 22 16:43:08.483351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.483077 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="thanos-sidecar" Jul 22 16:43:08.483351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.483083 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="kube-rbac-proxy-web" Jul 22 16:43:08.483351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.483088 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" containerName="config-reloader" Jul 22 16:43:08.483351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.483228 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2"} err="failed to get container status \"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2\": rpc error: code = NotFound desc = could not find container \"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2\": container with ID starting with 8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2 not found: ID does not exist" Jul 22 16:43:08.483351 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.483245 2557 scope.go:117] "RemoveContainer" containerID="452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81" Jul 22 16:43:08.483778 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.483441 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81"} err="failed to get container status \"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81\": rpc error: code = NotFound desc = could not find container \"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81\": container with ID starting with 452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81 not found: ID does not exist" Jul 22 16:43:08.483778 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.483457 2557 scope.go:117] "RemoveContainer" containerID="b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db" Jul 22 16:43:08.483778 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.483735 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db"} err="failed to get container status \"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db\": rpc error: code = NotFound desc = could not find container \"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db\": container with ID starting with b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db not found: ID does not exist" Jul 22 16:43:08.483778 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.483759 2557 scope.go:117] "RemoveContainer" containerID="7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1" Jul 22 16:43:08.484115 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.484087 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1"} err="failed to get container status \"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1\": rpc error: code = NotFound desc = could not find container \"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1\": container with ID starting with 7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1 not found: ID does not exist" Jul 22 16:43:08.484115 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.484114 2557 scope.go:117] "RemoveContainer" containerID="96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27" Jul 22 16:43:08.484406 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.484386 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27"} err="failed to get container status \"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27\": rpc error: code = NotFound desc = could not find container \"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27\": container with ID starting with 96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27 not found: ID does not exist" Jul 22 16:43:08.484406 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.484406 2557 scope.go:117] "RemoveContainer" containerID="ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c" Jul 22 16:43:08.484686 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.484662 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c"} err="failed to get container status \"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c\": rpc error: code = NotFound desc = could not find container \"ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c\": container with ID starting with ea4528309b33d1b19f7bd4d5486d84056a76c73a2bf5809c741757f7cc351b0c not found: ID does not exist" Jul 22 16:43:08.484754 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.484687 2557 scope.go:117] "RemoveContainer" containerID="b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025" Jul 22 16:43:08.484989 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.484966 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025"} err="failed to get container status \"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025\": rpc error: code = NotFound desc = could not find container \"b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025\": container with ID starting with b8ee91172016a3f9e8c156da8d09fdda98b27096c702f3ad8b7fbb3a3074b025 not found: ID does not exist" Jul 22 16:43:08.484989 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.484992 2557 scope.go:117] "RemoveContainer" containerID="8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2" Jul 22 16:43:08.485259 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.485236 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2"} err="failed to get container status \"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2\": rpc error: code = NotFound desc = could not find container \"8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2\": container with ID starting with 8b60f0787a3429f1c03b019e024bc212f9e258bdf95a5618c49f4c0376269fc2 not found: ID does not exist" Jul 22 16:43:08.485319 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.485261 2557 scope.go:117] "RemoveContainer" containerID="452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81" Jul 22 16:43:08.485507 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.485487 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81"} err="failed to get container status \"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81\": rpc error: code = NotFound desc = could not find container \"452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81\": container with ID starting with 452e33949e2100d1a2ac369d9b94221e5d9ea665ba6ab919aef858566e30ac81 not found: ID does not exist" Jul 22 16:43:08.485555 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.485507 2557 scope.go:117] "RemoveContainer" containerID="b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db" Jul 22 16:43:08.485751 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.485732 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db"} err="failed to get container status \"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db\": rpc error: code = NotFound desc = could not find container \"b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db\": container with ID starting with b88d27410fb679a1846c673fef3e96773b5b2159a882c5eda4f1c673b3fdb9db not found: ID does not exist" Jul 22 16:43:08.485751 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.485750 2557 scope.go:117] "RemoveContainer" containerID="7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1" Jul 22 16:43:08.485955 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.485940 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1"} err="failed to get container status \"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1\": rpc error: code = NotFound desc = could not find container \"7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1\": container with ID starting with 7c012c09f554590693551824ff31cf3840691c674f48e3c12ea8744fbc69e2d1 not found: ID does not exist" Jul 22 16:43:08.485998 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.485956 2557 scope.go:117] "RemoveContainer" containerID="96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27" Jul 22 16:43:08.486201 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.486186 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27"} err="failed to get container status \"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27\": rpc error: code = NotFound desc = could not find container \"96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27\": container with ID starting with 96decf984c2e8365f24759b47f7a9e9acf9540eb594e2abf5d78385ecab78e27 not found: ID does not exist" Jul 22 16:43:08.487100 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.487085 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.489956 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.489936 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 22 16:43:08.490074 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.489998 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 22 16:43:08.490074 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.489937 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sptlh\"" Jul 22 16:43:08.490074 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.490012 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 22 16:43:08.490241 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.490012 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 22 16:43:08.490379 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.490357 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 22 16:43:08.490462 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.490384 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 22 16:43:08.490462 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.490399 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fk2aeqp6g3ko8\"" Jul 22 16:43:08.490462 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.490370 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 22 16:43:08.490596 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.490462 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 22 16:43:08.490596 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.490488 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 22 16:43:08.490694 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.490679 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 22 16:43:08.490842 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.490828 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 22 16:43:08.491075 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.491062 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 22 16:43:08.492709 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.492689 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 22 16:43:08.496400 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.496383 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 22 16:43:08.506480 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.506458 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 16:43:08.623742 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.623659 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.623742 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.623690 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0eb3d5b7-040c-4232-bd15-54c61a468c0f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.623742 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.623714 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.623975 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.623747 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.623975 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.623813 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.623975 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.623839 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.623975 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.623865 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.623975 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.623914 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-config\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.623975 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.623932 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-web-config\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.623975 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.623948 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0eb3d5b7-040c-4232-bd15-54c61a468c0f-config-out\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.623975 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.623973 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.624234 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.623997 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.624234 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.624068 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.624234 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.624088 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.624234 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.624114 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.624234 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.624134 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.624234 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.624149 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0eb3d5b7-040c-4232-bd15-54c61a468c0f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.624234 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.624200 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.624234 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.624225 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njr2q\" (UniqueName: \"kubernetes.io/projected/0eb3d5b7-040c-4232-bd15-54c61a468c0f-kube-api-access-njr2q\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.624471 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.624252 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.724553 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724515 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.724553 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724557 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.724793 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724575 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0eb3d5b7-040c-4232-bd15-54c61a468c0f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.724793 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724597 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.724793 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724620 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njr2q\" (UniqueName: \"kubernetes.io/projected/0eb3d5b7-040c-4232-bd15-54c61a468c0f-kube-api-access-njr2q\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.724793 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724639 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.724793 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724661 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.724793 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724684 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0eb3d5b7-040c-4232-bd15-54c61a468c0f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.724793 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724722 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.724793 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724748 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.724793 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724779 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.725264 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724802 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.725264 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724824 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.725264 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724856 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-config\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.725264 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724883 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-web-config\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.725264 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724907 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0eb3d5b7-040c-4232-bd15-54c61a468c0f-config-out\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.725264 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724929 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.725264 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724953 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.725264 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.724997 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.725264 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.725052 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.725264 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.725100 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0eb3d5b7-040c-4232-bd15-54c61a468c0f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.725777 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.725343 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.726955 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.726097 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.726955 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.726452 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.726955 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.726493 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.727233 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.726954 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.729309 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.727580 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.729309 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.728053 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0eb3d5b7-040c-4232-bd15-54c61a468c0f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.729309 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.728461 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-config\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.729309 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.728584 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.729309 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.728831 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.729635 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.729473 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.729635 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.729569 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0eb3d5b7-040c-4232-bd15-54c61a468c0f-config-out\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.729886 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.729847 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.730034 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.730003 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.730512 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.730476 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.730745 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.730728 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-web-config\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.730809 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.730788 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0eb3d5b7-040c-4232-bd15-54c61a468c0f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.731104 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.731088 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0eb3d5b7-040c-4232-bd15-54c61a468c0f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.733570 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.733550 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njr2q\" (UniqueName: \"kubernetes.io/projected/0eb3d5b7-040c-4232-bd15-54c61a468c0f-kube-api-access-njr2q\") pod \"prometheus-k8s-0\" (UID: \"0eb3d5b7-040c-4232-bd15-54c61a468c0f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.798083 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.798056 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:08.937144 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:08.937121 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 16:43:08.939155 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:43:08.939130 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb3d5b7_040c_4232_bd15_54c61a468c0f.slice/crio-b6c211019710a3c67f0841f7aba329bc9ad61e8eb6ae9df70b3f1745657c758c WatchSource:0}: Error finding container b6c211019710a3c67f0841f7aba329bc9ad61e8eb6ae9df70b3f1745657c758c: Status 404 returned error can't find the container with id b6c211019710a3c67f0841f7aba329bc9ad61e8eb6ae9df70b3f1745657c758c Jul 22 16:43:09.436169 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:09.436135 2557 generic.go:356] "Generic (PLEG): container finished" podID="0eb3d5b7-040c-4232-bd15-54c61a468c0f" containerID="a5af7178906379a38afc397918714d6660564915dc8b9c74a416e4fc4acdf45b" exitCode=0 Jul 22 16:43:09.436558 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:09.436218 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eb3d5b7-040c-4232-bd15-54c61a468c0f","Type":"ContainerDied","Data":"a5af7178906379a38afc397918714d6660564915dc8b9c74a416e4fc4acdf45b"} Jul 22 16:43:09.436558 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:09.436310 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eb3d5b7-040c-4232-bd15-54c61a468c0f","Type":"ContainerStarted","Data":"b6c211019710a3c67f0841f7aba329bc9ad61e8eb6ae9df70b3f1745657c758c"} Jul 22 16:43:09.575945 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:09.575916 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="634e0201-7092-4ea1-b6b4-62feecebe160" path="/var/lib/kubelet/pods/634e0201-7092-4ea1-b6b4-62feecebe160/volumes" Jul 22 16:43:10.443920 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:10.443878 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eb3d5b7-040c-4232-bd15-54c61a468c0f","Type":"ContainerStarted","Data":"65ce7d4e3405c423ed1ea2168e3b35a55ea3837a3da63384c14065c3e6a21916"} Jul 22 16:43:10.444357 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:10.443944 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eb3d5b7-040c-4232-bd15-54c61a468c0f","Type":"ContainerStarted","Data":"99a6d488d34631a96ec03c9e2efc00e452ebb5ea6cca9f6784d25c48fb23c1d3"} Jul 22 16:43:10.444357 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:10.443984 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eb3d5b7-040c-4232-bd15-54c61a468c0f","Type":"ContainerStarted","Data":"7ba13586cb9d3847bf4835914fd6740962bbe25e95f08fa69c50748b2a51b81f"} Jul 22 16:43:10.444357 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:10.444013 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eb3d5b7-040c-4232-bd15-54c61a468c0f","Type":"ContainerStarted","Data":"fb134738f39ba072e7e38556fd01983fc587037f6862b027e7eb8b2e12b7116d"} Jul 22 16:43:10.444357 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:10.444072 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eb3d5b7-040c-4232-bd15-54c61a468c0f","Type":"ContainerStarted","Data":"b757bb2458630c96a818b5cf7536d77e7b131f0d7ac7f7ebba65fa95c8528113"} Jul 22 16:43:10.444357 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:10.444097 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0eb3d5b7-040c-4232-bd15-54c61a468c0f","Type":"ContainerStarted","Data":"316da5a31a3effab3e352afe8f42c29a1896dbc14f181be0222749fb3790bafd"} Jul 22 16:43:10.472786 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:10.472737 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.453569923 podStartE2EDuration="2.472722032s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.152109ms" isStatefulPod=false podCreationTimestamp="2026-07-22 16:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:43:10.470902069 +0000 UTC m=+155.447630434" watchObservedRunningTime="2026-07-22 16:43:10.472722032 +0000 UTC m=+155.449450348" Jul 22 16:43:12.703614 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:12.703579 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:12.803546 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:12.803507 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:12.855293 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:12.855260 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:13.384481 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:13.384447 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:13.798317 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:13.798286 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:43:23.662923 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:23.662891 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:23.726722 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:23.726690 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:23.788234 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:23.788198 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:41.159616 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:41.159578 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:43:41.261610 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:43:41.261578 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:44:08.798357 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:44:08.798328 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:44:08.813464 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:44:08.813436 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:44:09.688552 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:44:09.688521 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 16:44:17.414551 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:44:17.414512 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:44:17.504619 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:44:17.504584 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:44:17.576557 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:44:17.576520 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:44:18.079807 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:44:18.079775 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:44:18.132906 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:44:18.132872 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:44:28.883082 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:44:28.883044 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:44:28.937532 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:44:28.937499 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:44:28.998321 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:44:28.998285 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:44:29.370446 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:44:29.370388 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:44:29.428629 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:44:29.428580 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:44:39.094977 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:44:39.094944 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:44:39.194417 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:44:39.194333 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:45:35.418648 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:45:35.418618 2557 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 16:45:39.998749 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:45:39.998710 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:45:42.122899 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:45:42.122857 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:07.509735 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:07.509698 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:07.903152 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:07.903070 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:11.753123 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:11.753084 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:11.863666 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:11.863633 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:13.655341 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:13.655302 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:13.788987 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:13.788948 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:13.955263 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:13.955171 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:16.314390 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:16.314348 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:16.550167 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:16.550128 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:21.758780 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:21.758744 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jws7n"] Jul 22 16:46:21.761922 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:21.761906 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jws7n" Jul 22 16:46:21.766448 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:21.766429 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-47kp7\"" Jul 22 16:46:21.767387 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:21.767370 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 16:46:21.772065 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:21.772047 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 16:46:21.787648 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:21.787627 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jws7n"] Jul 22 16:46:21.820140 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:21.820111 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d3e26dd-2a51-4a7d-9136-151a018a67b6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jws7n\" (UID: \"9d3e26dd-2a51-4a7d-9136-151a018a67b6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jws7n" Jul 22 16:46:21.820274 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:21.820148 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zlrn\" (UniqueName: \"kubernetes.io/projected/9d3e26dd-2a51-4a7d-9136-151a018a67b6-kube-api-access-7zlrn\") pod \"cert-manager-webhook-6486ffd997-jws7n\" (UID: \"9d3e26dd-2a51-4a7d-9136-151a018a67b6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jws7n" Jul 22 16:46:21.920951 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:21.920916 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlrn\" (UniqueName: \"kubernetes.io/projected/9d3e26dd-2a51-4a7d-9136-151a018a67b6-kube-api-access-7zlrn\") pod \"cert-manager-webhook-6486ffd997-jws7n\" (UID: \"9d3e26dd-2a51-4a7d-9136-151a018a67b6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jws7n" Jul 22 16:46:21.921140 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:21.920997 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d3e26dd-2a51-4a7d-9136-151a018a67b6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jws7n\" (UID: \"9d3e26dd-2a51-4a7d-9136-151a018a67b6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jws7n" Jul 22 16:46:21.944370 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:21.944345 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d3e26dd-2a51-4a7d-9136-151a018a67b6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-jws7n\" (UID: \"9d3e26dd-2a51-4a7d-9136-151a018a67b6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jws7n" Jul 22 16:46:21.945603 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:21.945582 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zlrn\" (UniqueName: \"kubernetes.io/projected/9d3e26dd-2a51-4a7d-9136-151a018a67b6-kube-api-access-7zlrn\") pod \"cert-manager-webhook-6486ffd997-jws7n\" (UID: \"9d3e26dd-2a51-4a7d-9136-151a018a67b6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-jws7n" Jul 22 16:46:22.083602 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:22.083505 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-jws7n" Jul 22 16:46:22.220670 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:22.220639 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-jws7n"] Jul 22 16:46:22.223709 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:46:22.223687 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d3e26dd_2a51_4a7d_9136_151a018a67b6.slice/crio-e4cefeda760bceb21f4c2eb4c1b15a43151e9b035842a2f8043b3484d549d431 WatchSource:0}: Error finding container e4cefeda760bceb21f4c2eb4c1b15a43151e9b035842a2f8043b3484d549d431: Status 404 returned error can't find the container with id e4cefeda760bceb21f4c2eb4c1b15a43151e9b035842a2f8043b3484d549d431 Jul 22 16:46:22.225566 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:22.225550 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:46:23.193558 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:23.193520 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jws7n" event={"ID":"9d3e26dd-2a51-4a7d-9136-151a018a67b6","Type":"ContainerStarted","Data":"e4cefeda760bceb21f4c2eb4c1b15a43151e9b035842a2f8043b3484d549d431"} Jul 22 16:46:24.425440 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:24.425402 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:24.713361 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:24.713189 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:24.817341 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:24.817303 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:26.102275 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.102243 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddlg5"] Jul 22 16:46:26.105541 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.105524 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddlg5" Jul 22 16:46:26.110939 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.110918 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 16:46:26.111417 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.111401 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 16:46:26.112369 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.112353 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-mvf8c\"" Jul 22 16:46:26.130758 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.130737 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddlg5"] Jul 22 16:46:26.155410 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.155390 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlp9x\" (UniqueName: \"kubernetes.io/projected/b3d6f982-55c0-45a0-8ab6-8fdad5d2873c-kube-api-access-xlp9x\") pod \"openshift-lws-operator-fd8ccff4c-ddlg5\" (UID: \"b3d6f982-55c0-45a0-8ab6-8fdad5d2873c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddlg5" Jul 22 16:46:26.155533 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.155438 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3d6f982-55c0-45a0-8ab6-8fdad5d2873c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ddlg5\" (UID: \"b3d6f982-55c0-45a0-8ab6-8fdad5d2873c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddlg5" Jul 22 16:46:26.208465 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.208435 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-jws7n" event={"ID":"9d3e26dd-2a51-4a7d-9136-151a018a67b6","Type":"ContainerStarted","Data":"fd8bb5bb0359dda10f6608ff77fcb9a08121079a37e5c7a0084ba4c21db9e945"} Jul 22 16:46:26.208587 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.208558 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jws7n" Jul 22 16:46:26.255450 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.255358 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-jws7n" podStartSLOduration=1.4805212349999999 podStartE2EDuration="5.255344312s" totalImagesPullingTime="3.774823077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:46:26.242854388 +0000 UTC m=+351.219582706" watchObservedRunningTime="2026-07-22 16:46:26.255344312 +0000 UTC m=+351.232072627" Jul 22 16:46:26.256489 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.256442 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3d6f982-55c0-45a0-8ab6-8fdad5d2873c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ddlg5\" (UID: \"b3d6f982-55c0-45a0-8ab6-8fdad5d2873c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddlg5" Jul 22 16:46:26.256644 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.256565 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlp9x\" (UniqueName: \"kubernetes.io/projected/b3d6f982-55c0-45a0-8ab6-8fdad5d2873c-kube-api-access-xlp9x\") pod \"openshift-lws-operator-fd8ccff4c-ddlg5\" (UID: \"b3d6f982-55c0-45a0-8ab6-8fdad5d2873c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddlg5" Jul 22 16:46:26.256907 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.256886 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3d6f982-55c0-45a0-8ab6-8fdad5d2873c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ddlg5\" (UID: \"b3d6f982-55c0-45a0-8ab6-8fdad5d2873c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddlg5" Jul 22 16:46:26.288222 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.288194 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlp9x\" (UniqueName: \"kubernetes.io/projected/b3d6f982-55c0-45a0-8ab6-8fdad5d2873c-kube-api-access-xlp9x\") pod \"openshift-lws-operator-fd8ccff4c-ddlg5\" (UID: \"b3d6f982-55c0-45a0-8ab6-8fdad5d2873c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddlg5" Jul 22 16:46:26.414568 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.414536 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddlg5" Jul 22 16:46:26.557073 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:26.557048 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddlg5"] Jul 22 16:46:26.558643 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:46:26.558611 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3d6f982_55c0_45a0_8ab6_8fdad5d2873c.slice/crio-7097351e09b551a51f57d18bca637f166e5683c22681412e96e5f680a7194838 WatchSource:0}: Error finding container 7097351e09b551a51f57d18bca637f166e5683c22681412e96e5f680a7194838: Status 404 returned error can't find the container with id 7097351e09b551a51f57d18bca637f166e5683c22681412e96e5f680a7194838 Jul 22 16:46:27.213092 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:27.213057 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddlg5" event={"ID":"b3d6f982-55c0-45a0-8ab6-8fdad5d2873c","Type":"ContainerStarted","Data":"7097351e09b551a51f57d18bca637f166e5683c22681412e96e5f680a7194838"} Jul 22 16:46:30.234818 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:30.234779 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddlg5" event={"ID":"b3d6f982-55c0-45a0-8ab6-8fdad5d2873c","Type":"ContainerStarted","Data":"001a5c50a61ba8569cf632b5f297814ad664334de8f31088c393d37dfa097bb2"} Jul 22 16:46:30.268420 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:30.268366 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ddlg5" podStartSLOduration=0.738195394 podStartE2EDuration="4.268350402s" totalImagesPullingTime="3.530155008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:46:30.266281036 +0000 UTC m=+355.243009353" watchObservedRunningTime="2026-07-22 16:46:30.268350402 +0000 UTC m=+355.245078717" Jul 22 16:46:31.113844 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:31.113806 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:31.211457 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:31.211417 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:32.215863 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:32.215832 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-jws7n" Jul 22 16:46:33.325372 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:33.325341 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:51.398033 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:51.397994 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:51.689060 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:51.688955 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:51.802254 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:51.802219 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:51.991069 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:51.991034 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:53.146649 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:53.146615 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:55.655608 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:55.655579 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:55.938758 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:55.938675 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:56.127430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.127399 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j"] Jul 22 16:46:56.136868 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.136851 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.144596 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.144579 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 16:46:56.146252 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.146233 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4qzcs\"" Jul 22 16:46:56.146444 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.146425 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 16:46:56.146555 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.146515 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 16:46:56.146555 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.146551 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 16:46:56.181249 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.181225 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j"] Jul 22 16:46:56.215568 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.215491 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e326eb82-b59a-4638-a7e2-e12b03bee5a0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55b7bf7664-8gr5j\" (UID: \"e326eb82-b59a-4638-a7e2-e12b03bee5a0\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.215568 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.215543 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4x8h\" (UniqueName: \"kubernetes.io/projected/e326eb82-b59a-4638-a7e2-e12b03bee5a0-kube-api-access-w4x8h\") pod \"opendatahub-operator-controller-manager-55b7bf7664-8gr5j\" (UID: \"e326eb82-b59a-4638-a7e2-e12b03bee5a0\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.215782 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.215616 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e326eb82-b59a-4638-a7e2-e12b03bee5a0-manifests\") pod \"opendatahub-operator-controller-manager-55b7bf7664-8gr5j\" (UID: \"e326eb82-b59a-4638-a7e2-e12b03bee5a0\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.215782 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.215672 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e326eb82-b59a-4638-a7e2-e12b03bee5a0-webhook-cert\") pod \"opendatahub-operator-controller-manager-55b7bf7664-8gr5j\" (UID: \"e326eb82-b59a-4638-a7e2-e12b03bee5a0\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.215782 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.215714 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e326eb82-b59a-4638-a7e2-e12b03bee5a0-tmp\") pod \"opendatahub-operator-controller-manager-55b7bf7664-8gr5j\" (UID: \"e326eb82-b59a-4638-a7e2-e12b03bee5a0\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.316915 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.316870 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4x8h\" (UniqueName: \"kubernetes.io/projected/e326eb82-b59a-4638-a7e2-e12b03bee5a0-kube-api-access-w4x8h\") pod \"opendatahub-operator-controller-manager-55b7bf7664-8gr5j\" (UID: \"e326eb82-b59a-4638-a7e2-e12b03bee5a0\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.316915 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.316909 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e326eb82-b59a-4638-a7e2-e12b03bee5a0-manifests\") pod \"opendatahub-operator-controller-manager-55b7bf7664-8gr5j\" (UID: \"e326eb82-b59a-4638-a7e2-e12b03bee5a0\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.317164 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.316932 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e326eb82-b59a-4638-a7e2-e12b03bee5a0-webhook-cert\") pod \"opendatahub-operator-controller-manager-55b7bf7664-8gr5j\" (UID: \"e326eb82-b59a-4638-a7e2-e12b03bee5a0\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.317164 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.316956 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e326eb82-b59a-4638-a7e2-e12b03bee5a0-tmp\") pod \"opendatahub-operator-controller-manager-55b7bf7664-8gr5j\" (UID: \"e326eb82-b59a-4638-a7e2-e12b03bee5a0\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.317164 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.317013 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e326eb82-b59a-4638-a7e2-e12b03bee5a0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55b7bf7664-8gr5j\" (UID: \"e326eb82-b59a-4638-a7e2-e12b03bee5a0\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.317333 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.317313 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e326eb82-b59a-4638-a7e2-e12b03bee5a0-manifests\") pod \"opendatahub-operator-controller-manager-55b7bf7664-8gr5j\" (UID: \"e326eb82-b59a-4638-a7e2-e12b03bee5a0\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.319244 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.319216 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e326eb82-b59a-4638-a7e2-e12b03bee5a0-tmp\") pod \"opendatahub-operator-controller-manager-55b7bf7664-8gr5j\" (UID: \"e326eb82-b59a-4638-a7e2-e12b03bee5a0\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.319473 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.319451 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e326eb82-b59a-4638-a7e2-e12b03bee5a0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55b7bf7664-8gr5j\" (UID: \"e326eb82-b59a-4638-a7e2-e12b03bee5a0\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.319574 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.319556 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e326eb82-b59a-4638-a7e2-e12b03bee5a0-webhook-cert\") pod \"opendatahub-operator-controller-manager-55b7bf7664-8gr5j\" (UID: \"e326eb82-b59a-4638-a7e2-e12b03bee5a0\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.352430 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.352402 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4x8h\" (UniqueName: \"kubernetes.io/projected/e326eb82-b59a-4638-a7e2-e12b03bee5a0-kube-api-access-w4x8h\") pod \"opendatahub-operator-controller-manager-55b7bf7664-8gr5j\" (UID: \"e326eb82-b59a-4638-a7e2-e12b03bee5a0\") " pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.400339 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.400306 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:56.446295 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.446267 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:46:56.520290 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.520254 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:56.636320 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.636243 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j"] Jul 22 16:46:56.639248 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:46:56.639222 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode326eb82_b59a_4638_a7e2_e12b03bee5a0.slice/crio-ac4145b6db7b6c40aefafb35d3ca626200815366ba9fb5c60b0f539eac19325b WatchSource:0}: Error finding container ac4145b6db7b6c40aefafb35d3ca626200815366ba9fb5c60b0f539eac19325b: Status 404 returned error can't find the container with id ac4145b6db7b6c40aefafb35d3ca626200815366ba9fb5c60b0f539eac19325b Jul 22 16:46:56.672078 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.672046 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:56.795867 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.795783 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:56.905957 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:56.905917 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:57.039104 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:57.039067 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:57.148840 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:57.148731 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:57.249247 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:57.249198 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:57.350705 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:57.350663 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" event={"ID":"e326eb82-b59a-4638-a7e2-e12b03bee5a0","Type":"ContainerStarted","Data":"ac4145b6db7b6c40aefafb35d3ca626200815366ba9fb5c60b0f539eac19325b"} Jul 22 16:46:58.218159 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:58.218113 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:46:58.386900 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:46:58.386826 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:00.366893 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:00.366848 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" event={"ID":"e326eb82-b59a-4638-a7e2-e12b03bee5a0","Type":"ContainerStarted","Data":"4210671debfa419b7c63023d94fb1b5cdca73e245d60dfa8fa5a20f849856919"} Jul 22 16:47:01.372630 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:01.372592 2557 generic.go:356] "Generic (PLEG): container finished" podID="e326eb82-b59a-4638-a7e2-e12b03bee5a0" containerID="4210671debfa419b7c63023d94fb1b5cdca73e245d60dfa8fa5a20f849856919" exitCode=0 Jul 22 16:47:01.373060 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:01.372655 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" event={"ID":"e326eb82-b59a-4638-a7e2-e12b03bee5a0","Type":"ContainerDied","Data":"4210671debfa419b7c63023d94fb1b5cdca73e245d60dfa8fa5a20f849856919"} Jul 22 16:47:02.378863 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:02.378820 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" event={"ID":"e326eb82-b59a-4638-a7e2-e12b03bee5a0","Type":"ContainerStarted","Data":"2067e40263004eb07fb52bce79f085d78a2e4dc0cb0d08aae6227963dc6a5006"} Jul 22 16:47:02.379266 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:02.379037 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:47:02.469702 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:02.469652 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" podStartSLOduration=2.078572359 podStartE2EDuration="6.469635043s" totalImagesPullingTime="3.369224813s" totalInitContainerRuntime="1.021837871s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:46:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:47:02.466562238 +0000 UTC m=+387.443290579" watchObservedRunningTime="2026-07-22 16:47:02.469635043 +0000 UTC m=+387.446363358" Jul 22 16:47:04.587674 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:04.587638 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:04.750597 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:04.750563 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:13.385931 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:13.385902 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55b7bf7664-8gr5j" Jul 22 16:47:16.698683 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:16.698646 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:16.861185 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:16.861150 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:17.024349 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:17.024306 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:17.621144 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:17.621107 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:17.953321 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:17.953229 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:18.232441 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:18.232402 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:18.379567 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:18.379532 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:18.688070 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:18.687976 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:18.981711 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:18.981669 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:20.402035 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:20.401998 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:20.975744 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:20.975700 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:22.247768 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:22.247736 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:22.380445 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:22.380407 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:22.502288 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:22.502205 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:25.297437 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:25.297388 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:26.965563 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:26.965528 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq"] Jul 22 16:47:26.973755 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:26.973732 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:26.983127 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:26.983107 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 22 16:47:26.983225 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:26.983161 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 22 16:47:26.983225 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:26.983184 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wvk7r\"" Jul 22 16:47:26.983225 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:26.983195 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 22 16:47:26.987692 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:26.987672 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq"] Jul 22 16:47:27.091679 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.091639 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/783ac916-7082-4f82-a3bd-a0a1630be59e-cert\") pod \"lws-controller-manager-57844c55d9-dc7mq\" (UID: \"783ac916-7082-4f82-a3bd-a0a1630be59e\") " pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:27.091853 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.091696 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/783ac916-7082-4f82-a3bd-a0a1630be59e-manager-config\") pod \"lws-controller-manager-57844c55d9-dc7mq\" (UID: \"783ac916-7082-4f82-a3bd-a0a1630be59e\") " pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:27.091853 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.091716 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd6zn\" (UniqueName: \"kubernetes.io/projected/783ac916-7082-4f82-a3bd-a0a1630be59e-kube-api-access-bd6zn\") pod \"lws-controller-manager-57844c55d9-dc7mq\" (UID: \"783ac916-7082-4f82-a3bd-a0a1630be59e\") " pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:27.091853 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.091764 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/783ac916-7082-4f82-a3bd-a0a1630be59e-metrics-cert\") pod \"lws-controller-manager-57844c55d9-dc7mq\" (UID: \"783ac916-7082-4f82-a3bd-a0a1630be59e\") " pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:27.192833 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.192799 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/783ac916-7082-4f82-a3bd-a0a1630be59e-manager-config\") pod \"lws-controller-manager-57844c55d9-dc7mq\" (UID: \"783ac916-7082-4f82-a3bd-a0a1630be59e\") " pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:27.192833 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.192835 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd6zn\" (UniqueName: \"kubernetes.io/projected/783ac916-7082-4f82-a3bd-a0a1630be59e-kube-api-access-bd6zn\") pod \"lws-controller-manager-57844c55d9-dc7mq\" (UID: \"783ac916-7082-4f82-a3bd-a0a1630be59e\") " pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:27.193061 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.192885 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/783ac916-7082-4f82-a3bd-a0a1630be59e-metrics-cert\") pod \"lws-controller-manager-57844c55d9-dc7mq\" (UID: \"783ac916-7082-4f82-a3bd-a0a1630be59e\") " pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:27.193061 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.192945 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/783ac916-7082-4f82-a3bd-a0a1630be59e-cert\") pod \"lws-controller-manager-57844c55d9-dc7mq\" (UID: \"783ac916-7082-4f82-a3bd-a0a1630be59e\") " pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:27.193593 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.193570 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/783ac916-7082-4f82-a3bd-a0a1630be59e-manager-config\") pod \"lws-controller-manager-57844c55d9-dc7mq\" (UID: \"783ac916-7082-4f82-a3bd-a0a1630be59e\") " pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:27.195357 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.195334 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/783ac916-7082-4f82-a3bd-a0a1630be59e-metrics-cert\") pod \"lws-controller-manager-57844c55d9-dc7mq\" (UID: \"783ac916-7082-4f82-a3bd-a0a1630be59e\") " pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:27.195456 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.195381 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/783ac916-7082-4f82-a3bd-a0a1630be59e-cert\") pod \"lws-controller-manager-57844c55d9-dc7mq\" (UID: \"783ac916-7082-4f82-a3bd-a0a1630be59e\") " pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:27.206653 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.206633 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd6zn\" (UniqueName: \"kubernetes.io/projected/783ac916-7082-4f82-a3bd-a0a1630be59e-kube-api-access-bd6zn\") pod \"lws-controller-manager-57844c55d9-dc7mq\" (UID: \"783ac916-7082-4f82-a3bd-a0a1630be59e\") " pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:27.282469 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.282446 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:27.463384 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.463358 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq"] Jul 22 16:47:27.464247 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:47:27.464214 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783ac916_7082_4f82_a3bd_a0a1630be59e.slice/crio-5dae1f2bdf4b9ef097c7031bb3f9d81cc13009e4e6e28c8b66df4845c012c37e WatchSource:0}: Error finding container 5dae1f2bdf4b9ef097c7031bb3f9d81cc13009e4e6e28c8b66df4845c012c37e: Status 404 returned error can't find the container with id 5dae1f2bdf4b9ef097c7031bb3f9d81cc13009e4e6e28c8b66df4845c012c37e Jul 22 16:47:27.494572 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.494540 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" event={"ID":"783ac916-7082-4f82-a3bd-a0a1630be59e","Type":"ContainerStarted","Data":"5dae1f2bdf4b9ef097c7031bb3f9d81cc13009e4e6e28c8b66df4845c012c37e"} Jul 22 16:47:27.846915 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:27.846885 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:28.043871 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.039435 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:28.350097 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.350056 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l"] Jul 22 16:47:28.353496 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.353478 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.358698 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.358671 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 16:47:28.358698 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.358682 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 16:47:28.359149 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.358866 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rthqn\"" Jul 22 16:47:28.359645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.359625 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 16:47:28.359851 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.359828 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 16:47:28.367377 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.367358 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 16:47:28.388289 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.388263 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 16:47:28.392124 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.392101 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l"] Jul 22 16:47:28.404217 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.404187 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1e71384d-5c47-4a2a-992c-c581ddf3669c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.404349 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.404225 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e71384d-5c47-4a2a-992c-c581ddf3669c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.404349 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.404253 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e71384d-5c47-4a2a-992c-c581ddf3669c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.404349 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.404299 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e71384d-5c47-4a2a-992c-c581ddf3669c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.404349 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.404333 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e71384d-5c47-4a2a-992c-c581ddf3669c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.404742 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.404358 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8z4v\" (UniqueName: \"kubernetes.io/projected/1e71384d-5c47-4a2a-992c-c581ddf3669c-kube-api-access-p8z4v\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.404742 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.404406 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e71384d-5c47-4a2a-992c-c581ddf3669c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.505489 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.505456 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e71384d-5c47-4a2a-992c-c581ddf3669c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.505679 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.505582 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1e71384d-5c47-4a2a-992c-c581ddf3669c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.505679 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.505609 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e71384d-5c47-4a2a-992c-c581ddf3669c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.505679 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.505631 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e71384d-5c47-4a2a-992c-c581ddf3669c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.505846 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.505694 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e71384d-5c47-4a2a-992c-c581ddf3669c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.505846 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.505730 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e71384d-5c47-4a2a-992c-c581ddf3669c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.505846 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.505759 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8z4v\" (UniqueName: \"kubernetes.io/projected/1e71384d-5c47-4a2a-992c-c581ddf3669c-kube-api-access-p8z4v\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.506663 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.506629 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1e71384d-5c47-4a2a-992c-c581ddf3669c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.508410 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.508376 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e71384d-5c47-4a2a-992c-c581ddf3669c-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.508504 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.508411 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1e71384d-5c47-4a2a-992c-c581ddf3669c-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.508504 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.508478 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1e71384d-5c47-4a2a-992c-c581ddf3669c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.508667 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.508645 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1e71384d-5c47-4a2a-992c-c581ddf3669c-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.553839 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.553802 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e71384d-5c47-4a2a-992c-c581ddf3669c-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.565707 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.565677 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8z4v\" (UniqueName: \"kubernetes.io/projected/1e71384d-5c47-4a2a-992c-c581ddf3669c-kube-api-access-p8z4v\") pod \"istiod-openshift-gateway-94bb8fbfd-gch7l\" (UID: \"1e71384d-5c47-4a2a-992c-c581ddf3669c\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.664797 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.664712 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:28.866285 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:28.866255 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l"] Jul 22 16:47:29.079306 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:47:29.079265 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e71384d_5c47_4a2a_992c_c581ddf3669c.slice/crio-69ca94666dbc768f16a740b02f79c22d4d44e842bf0c3a524156fd9dfc6924a4 WatchSource:0}: Error finding container 69ca94666dbc768f16a740b02f79c22d4d44e842bf0c3a524156fd9dfc6924a4: Status 404 returned error can't find the container with id 69ca94666dbc768f16a740b02f79c22d4d44e842bf0c3a524156fd9dfc6924a4 Jul 22 16:47:29.508434 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:29.508383 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" event={"ID":"1e71384d-5c47-4a2a-992c-c581ddf3669c","Type":"ContainerStarted","Data":"69ca94666dbc768f16a740b02f79c22d4d44e842bf0c3a524156fd9dfc6924a4"} Jul 22 16:47:29.510562 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:29.510442 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" event={"ID":"783ac916-7082-4f82-a3bd-a0a1630be59e","Type":"ContainerStarted","Data":"67aa0f139aa048301c68e7df1f128b5c75dbf7db4e8d6d8baa454beae1e2bd36"} Jul 22 16:47:29.510752 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:29.510596 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:29.551039 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:29.550967 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" podStartSLOduration=1.8796983969999999 podStartE2EDuration="3.55094702s" totalImagesPullingTime="1.671248623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:47:29.549457417 +0000 UTC m=+414.526185760" watchObservedRunningTime="2026-07-22 16:47:29.55094702 +0000 UTC m=+414.527675339" Jul 22 16:47:30.082873 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:30.082827 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:30.409440 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:30.409289 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:30.634637 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:30.634589 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:31.249846 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:31.249815 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 16:47:31.250131 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:31.249881 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 22 16:47:31.523669 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:31.523579 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" event={"ID":"1e71384d-5c47-4a2a-992c-c581ddf3669c","Type":"ContainerStarted","Data":"6e447a975bc7a798461899a5feb8040a7258f65c30c61f1c5308abe507eedc0d"} Jul 22 16:47:31.523840 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:31.523737 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:32.529849 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:32.529821 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" Jul 22 16:47:32.644404 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:32.644350 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gch7l" podStartSLOduration=2.476082753 podStartE2EDuration="4.644334755s" totalImagesPullingTime="2.168252002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:47:31.624044191 +0000 UTC m=+416.600772500" watchObservedRunningTime="2026-07-22 16:47:32.644334755 +0000 UTC m=+417.621063071" Jul 22 16:47:33.207495 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:33.207454 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:33.368987 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:33.368949 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:33.491031 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:33.490993 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:33.873135 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:33.873038 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:33.983695 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:33.983657 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:34.150533 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:34.150441 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:34.288497 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:34.288460 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:40.519692 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:40.519663 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57844c55d9-dc7mq" Jul 22 16:47:44.417314 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:44.417279 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:44.554171 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:44.554136 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:51.202034 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:51.201981 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:51.256659 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:51.256623 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:47:53.153478 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:47:53.153445 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:13.964214 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:13.964176 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:14.062461 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:14.062414 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:14.162835 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:14.162800 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:14.697115 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:14.697008 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:14.757788 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:14.757747 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:18.298751 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:18.298710 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:18.397827 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:18.397770 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:18.501733 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:18.501700 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:19.246996 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:19.246955 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:19.456916 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:19.456877 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:24.965310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:24.965269 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:25.067071 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:25.067013 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:25.165710 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:25.165678 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:47.691651 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:47.691606 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:51.871145 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:51.871108 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:54.327056 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:54.326996 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:54.481327 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:54.481283 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:55.747231 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:55.747198 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:58.893628 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:58.893593 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:59.279965 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:59.279928 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:59.625420 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:59.625335 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:48:59.773846 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:48:59.773813 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:02.329182 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:02.329149 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:03.687372 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:03.687340 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:03.793005 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:03.792970 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:07.481810 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:07.481770 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:07.582460 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:07.582389 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:08.476518 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:08.476472 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:08.630794 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:08.630762 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:10.872678 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:10.872638 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:10.941019 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:10.940982 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:18.188438 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.188389 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc"] Jul 22 16:49:18.191175 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.191156 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc" Jul 22 16:49:18.196275 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.196252 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ctrlr\"" Jul 22 16:49:18.196383 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.196296 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 16:49:18.196492 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.196469 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 16:49:18.206115 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.206093 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc"] Jul 22 16:49:18.275737 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.275705 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40104b83-e12a-480c-95fc-569835ec64bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jvsmc\" (UID: \"40104b83-e12a-480c-95fc-569835ec64bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc" Jul 22 16:49:18.275900 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.275743 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmxgg\" (UniqueName: \"kubernetes.io/projected/40104b83-e12a-480c-95fc-569835ec64bb-kube-api-access-dmxgg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jvsmc\" (UID: \"40104b83-e12a-480c-95fc-569835ec64bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc" Jul 22 16:49:18.376241 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.376206 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40104b83-e12a-480c-95fc-569835ec64bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jvsmc\" (UID: \"40104b83-e12a-480c-95fc-569835ec64bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc" Jul 22 16:49:18.376241 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.376244 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmxgg\" (UniqueName: \"kubernetes.io/projected/40104b83-e12a-480c-95fc-569835ec64bb-kube-api-access-dmxgg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jvsmc\" (UID: \"40104b83-e12a-480c-95fc-569835ec64bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc" Jul 22 16:49:18.376560 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.376538 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40104b83-e12a-480c-95fc-569835ec64bb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jvsmc\" (UID: \"40104b83-e12a-480c-95fc-569835ec64bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc" Jul 22 16:49:18.392907 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.392879 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmxgg\" (UniqueName: \"kubernetes.io/projected/40104b83-e12a-480c-95fc-569835ec64bb-kube-api-access-dmxgg\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-jvsmc\" (UID: \"40104b83-e12a-480c-95fc-569835ec64bb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc" Jul 22 16:49:18.501175 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.501149 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc" Jul 22 16:49:18.643046 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.643009 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc"] Jul 22 16:49:18.645560 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:49:18.645530 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40104b83_e12a_480c_95fc_569835ec64bb.slice/crio-bf5ed7716f9939d460232ab6b43006836b8de0b4f66f3b78b19458cf669ea04e WatchSource:0}: Error finding container bf5ed7716f9939d460232ab6b43006836b8de0b4f66f3b78b19458cf669ea04e: Status 404 returned error can't find the container with id bf5ed7716f9939d460232ab6b43006836b8de0b4f66f3b78b19458cf669ea04e Jul 22 16:49:18.869564 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.869491 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc"] Jul 22 16:49:18.877909 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.877886 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc"] Jul 22 16:49:18.892771 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.892739 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45"] Jul 22 16:49:18.897136 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.897116 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" Jul 22 16:49:18.911007 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.910983 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45"] Jul 22 16:49:18.980920 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.980848 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd42cc2f-a202-4f27-abb5-9a95044001a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7jq45\" (UID: \"bd42cc2f-a202-4f27-abb5-9a95044001a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" Jul 22 16:49:18.981121 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:18.980936 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vsn6\" (UniqueName: \"kubernetes.io/projected/bd42cc2f-a202-4f27-abb5-9a95044001a9-kube-api-access-2vsn6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7jq45\" (UID: \"bd42cc2f-a202-4f27-abb5-9a95044001a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" Jul 22 16:49:19.081482 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:19.081444 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd42cc2f-a202-4f27-abb5-9a95044001a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7jq45\" (UID: \"bd42cc2f-a202-4f27-abb5-9a95044001a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" Jul 22 16:49:19.081635 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:19.081497 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vsn6\" (UniqueName: \"kubernetes.io/projected/bd42cc2f-a202-4f27-abb5-9a95044001a9-kube-api-access-2vsn6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7jq45\" (UID: \"bd42cc2f-a202-4f27-abb5-9a95044001a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" Jul 22 16:49:19.081826 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:19.081802 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd42cc2f-a202-4f27-abb5-9a95044001a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7jq45\" (UID: \"bd42cc2f-a202-4f27-abb5-9a95044001a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" Jul 22 16:49:19.089923 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:19.089896 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vsn6\" (UniqueName: \"kubernetes.io/projected/bd42cc2f-a202-4f27-abb5-9a95044001a9-kube-api-access-2vsn6\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-7jq45\" (UID: \"bd42cc2f-a202-4f27-abb5-9a95044001a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" Jul 22 16:49:19.208735 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:19.208658 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" Jul 22 16:49:19.333992 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:19.333947 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:19.351195 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:19.351166 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45"] Jul 22 16:49:19.361947 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:49:19.361915 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd42cc2f_a202_4f27_abb5_9a95044001a9.slice/crio-62fd1cc25788873ed894a25537e4c85b6141bdce554da1537390cea024635033 WatchSource:0}: Error finding container 62fd1cc25788873ed894a25537e4c85b6141bdce554da1537390cea024635033: Status 404 returned error can't find the container with id 62fd1cc25788873ed894a25537e4c85b6141bdce554da1537390cea024635033 Jul 22 16:49:19.432043 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:19.431993 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:20.007324 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:20.007289 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" event={"ID":"bd42cc2f-a202-4f27-abb5-9a95044001a9","Type":"ContainerStarted","Data":"62fd1cc25788873ed894a25537e4c85b6141bdce554da1537390cea024635033"} Jul 22 16:49:21.975305 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:21.975245 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:22.081984 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:22.081938 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:23.025074 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:23.024955 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" event={"ID":"bd42cc2f-a202-4f27-abb5-9a95044001a9","Type":"ContainerStarted","Data":"aff6f56461a48d2c5b52bd41f39af4e9209b810e9e533b79be5ab063b25f3f01"} Jul 22 16:49:23.025074 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:23.025038 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" Jul 22 16:49:23.026614 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:23.026588 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc" podUID="40104b83-e12a-480c-95fc-569835ec64bb" containerName="manager" containerID="cri-o://5b139cc22174cfbfc9b75b36f1260dee2c7792edfd61e98d5ce473ec2e755c99" gracePeriod=2 Jul 22 16:49:23.046636 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:23.046601 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" podStartSLOduration=1.6355660319999998 podStartE2EDuration="5.046586318s" totalImagesPullingTime="3.411020286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:49:23.045621263 +0000 UTC m=+528.022349580" watchObservedRunningTime="2026-07-22 16:49:23.046586318 +0000 UTC m=+528.023314627" Jul 22 16:49:23.270724 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:23.270701 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc" Jul 22 16:49:23.272947 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:49:23.272923 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-jvsmc\" is forbidden: User \"system:node:ip-10-0-132-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-94.ec2.internal' and this object" podUID="40104b83-e12a-480c-95fc-569835ec64bb" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc" Jul 22 16:49:23.322442 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:23.322386 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40104b83-e12a-480c-95fc-569835ec64bb-extensions-socket-volume\") pod \"40104b83-e12a-480c-95fc-569835ec64bb\" (UID: \"40104b83-e12a-480c-95fc-569835ec64bb\") " Jul 22 16:49:23.322535 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:23.322483 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmxgg\" (UniqueName: \"kubernetes.io/projected/40104b83-e12a-480c-95fc-569835ec64bb-kube-api-access-dmxgg\") pod \"40104b83-e12a-480c-95fc-569835ec64bb\" (UID: \"40104b83-e12a-480c-95fc-569835ec64bb\") " Jul 22 16:49:23.322667 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:23.322632 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40104b83-e12a-480c-95fc-569835ec64bb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "40104b83-e12a-480c-95fc-569835ec64bb" (UID: "40104b83-e12a-480c-95fc-569835ec64bb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:49:23.324577 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:23.324553 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40104b83-e12a-480c-95fc-569835ec64bb-kube-api-access-dmxgg" (OuterVolumeSpecName: "kube-api-access-dmxgg") pod "40104b83-e12a-480c-95fc-569835ec64bb" (UID: "40104b83-e12a-480c-95fc-569835ec64bb"). InnerVolumeSpecName "kube-api-access-dmxgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:49:23.423905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:23.423876 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmxgg\" (UniqueName: \"kubernetes.io/projected/40104b83-e12a-480c-95fc-569835ec64bb-kube-api-access-dmxgg\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:49:23.423905 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:23.423904 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40104b83-e12a-480c-95fc-569835ec64bb-extensions-socket-volume\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:49:23.575251 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:23.575167 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40104b83-e12a-480c-95fc-569835ec64bb" path="/var/lib/kubelet/pods/40104b83-e12a-480c-95fc-569835ec64bb/volumes" Jul 22 16:49:24.032348 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:24.032312 2557 generic.go:356] "Generic (PLEG): container finished" podID="40104b83-e12a-480c-95fc-569835ec64bb" containerID="5b139cc22174cfbfc9b75b36f1260dee2c7792edfd61e98d5ce473ec2e755c99" exitCode=2 Jul 22 16:49:24.032348 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:24.032351 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc" Jul 22 16:49:24.032808 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:24.032409 2557 scope.go:117] "RemoveContainer" containerID="5b139cc22174cfbfc9b75b36f1260dee2c7792edfd61e98d5ce473ec2e755c99" Jul 22 16:49:24.037193 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:49:24.037168 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-jvsmc\" is forbidden: User \"system:node:ip-10-0-132-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-94.ec2.internal' and this object" podUID="40104b83-e12a-480c-95fc-569835ec64bb" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc" Jul 22 16:49:24.040876 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:24.040862 2557 scope.go:117] "RemoveContainer" containerID="5b139cc22174cfbfc9b75b36f1260dee2c7792edfd61e98d5ce473ec2e755c99" Jul 22 16:49:24.041150 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:49:24.041133 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b139cc22174cfbfc9b75b36f1260dee2c7792edfd61e98d5ce473ec2e755c99\": container with ID starting with 5b139cc22174cfbfc9b75b36f1260dee2c7792edfd61e98d5ce473ec2e755c99 not found: ID does not exist" containerID="5b139cc22174cfbfc9b75b36f1260dee2c7792edfd61e98d5ce473ec2e755c99" Jul 22 16:49:24.041200 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:24.041157 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b139cc22174cfbfc9b75b36f1260dee2c7792edfd61e98d5ce473ec2e755c99"} err="failed to get container status \"5b139cc22174cfbfc9b75b36f1260dee2c7792edfd61e98d5ce473ec2e755c99\": rpc error: code = NotFound desc = could not find container \"5b139cc22174cfbfc9b75b36f1260dee2c7792edfd61e98d5ce473ec2e755c99\": container with ID starting with 5b139cc22174cfbfc9b75b36f1260dee2c7792edfd61e98d5ce473ec2e755c99 not found: ID does not exist" Jul 22 16:49:24.778486 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:24.778457 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:24.887907 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:24.887873 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:25.573986 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:49:25.573945 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-jvsmc\" is forbidden: User \"system:node:ip-10-0-132-94.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-94.ec2.internal' and this object" podUID="40104b83-e12a-480c-95fc-569835ec64bb" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-jvsmc" Jul 22 16:49:28.371171 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:28.371076 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:28.472487 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:28.472450 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:34.035553 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:34.035518 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" Jul 22 16:49:37.215402 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:37.215370 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:37.315103 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:37.315070 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:39.822874 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:39.822840 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:39.923180 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:39.923116 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:43.018216 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:43.018174 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:43.119357 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:43.119316 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:46.357681 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:46.357637 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45"] Jul 22 16:49:46.358124 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:46.357907 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" podUID="bd42cc2f-a202-4f27-abb5-9a95044001a9" containerName="manager" containerID="cri-o://aff6f56461a48d2c5b52bd41f39af4e9209b810e9e533b79be5ab063b25f3f01" gracePeriod=10 Jul 22 16:49:46.423649 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:46.423613 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:46.527746 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:46.527621 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:46.601572 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:46.601551 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" Jul 22 16:49:46.730729 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:46.730702 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd42cc2f-a202-4f27-abb5-9a95044001a9-extensions-socket-volume\") pod \"bd42cc2f-a202-4f27-abb5-9a95044001a9\" (UID: \"bd42cc2f-a202-4f27-abb5-9a95044001a9\") " Jul 22 16:49:46.730889 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:46.730771 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vsn6\" (UniqueName: \"kubernetes.io/projected/bd42cc2f-a202-4f27-abb5-9a95044001a9-kube-api-access-2vsn6\") pod \"bd42cc2f-a202-4f27-abb5-9a95044001a9\" (UID: \"bd42cc2f-a202-4f27-abb5-9a95044001a9\") " Jul 22 16:49:46.731156 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:46.731129 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd42cc2f-a202-4f27-abb5-9a95044001a9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bd42cc2f-a202-4f27-abb5-9a95044001a9" (UID: "bd42cc2f-a202-4f27-abb5-9a95044001a9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 16:49:46.732844 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:46.732813 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd42cc2f-a202-4f27-abb5-9a95044001a9-kube-api-access-2vsn6" (OuterVolumeSpecName: "kube-api-access-2vsn6") pod "bd42cc2f-a202-4f27-abb5-9a95044001a9" (UID: "bd42cc2f-a202-4f27-abb5-9a95044001a9"). InnerVolumeSpecName "kube-api-access-2vsn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:49:46.832285 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:46.832244 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bd42cc2f-a202-4f27-abb5-9a95044001a9-extensions-socket-volume\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:49:46.832285 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:46.832277 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vsn6\" (UniqueName: \"kubernetes.io/projected/bd42cc2f-a202-4f27-abb5-9a95044001a9-kube-api-access-2vsn6\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:49:47.142348 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:47.142251 2557 generic.go:356] "Generic (PLEG): container finished" podID="bd42cc2f-a202-4f27-abb5-9a95044001a9" containerID="aff6f56461a48d2c5b52bd41f39af4e9209b810e9e533b79be5ab063b25f3f01" exitCode=0 Jul 22 16:49:47.142348 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:47.142296 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" Jul 22 16:49:47.142348 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:47.142334 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" event={"ID":"bd42cc2f-a202-4f27-abb5-9a95044001a9","Type":"ContainerDied","Data":"aff6f56461a48d2c5b52bd41f39af4e9209b810e9e533b79be5ab063b25f3f01"} Jul 22 16:49:47.142579 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:47.142416 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45" event={"ID":"bd42cc2f-a202-4f27-abb5-9a95044001a9","Type":"ContainerDied","Data":"62fd1cc25788873ed894a25537e4c85b6141bdce554da1537390cea024635033"} Jul 22 16:49:47.142579 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:47.142439 2557 scope.go:117] "RemoveContainer" containerID="aff6f56461a48d2c5b52bd41f39af4e9209b810e9e533b79be5ab063b25f3f01" Jul 22 16:49:47.151843 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:47.151820 2557 scope.go:117] "RemoveContainer" containerID="aff6f56461a48d2c5b52bd41f39af4e9209b810e9e533b79be5ab063b25f3f01" Jul 22 16:49:47.152164 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:49:47.152142 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff6f56461a48d2c5b52bd41f39af4e9209b810e9e533b79be5ab063b25f3f01\": container with ID starting with aff6f56461a48d2c5b52bd41f39af4e9209b810e9e533b79be5ab063b25f3f01 not found: ID does not exist" containerID="aff6f56461a48d2c5b52bd41f39af4e9209b810e9e533b79be5ab063b25f3f01" Jul 22 16:49:47.152252 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:47.152172 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff6f56461a48d2c5b52bd41f39af4e9209b810e9e533b79be5ab063b25f3f01"} err="failed to get container status \"aff6f56461a48d2c5b52bd41f39af4e9209b810e9e533b79be5ab063b25f3f01\": rpc error: code = NotFound desc = could not find container \"aff6f56461a48d2c5b52bd41f39af4e9209b810e9e533b79be5ab063b25f3f01\": container with ID starting with aff6f56461a48d2c5b52bd41f39af4e9209b810e9e533b79be5ab063b25f3f01 not found: ID does not exist" Jul 22 16:49:47.166583 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:47.166554 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45"] Jul 22 16:49:47.174879 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:47.174854 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-7jq45"] Jul 22 16:49:47.574463 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:47.574427 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd42cc2f-a202-4f27-abb5-9a95044001a9" path="/var/lib/kubelet/pods/bd42cc2f-a202-4f27-abb5-9a95044001a9/volumes" Jul 22 16:49:48.445310 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:48.445272 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:50.465822 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:50.465785 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:50.595141 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:50.595106 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:55.030458 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:55.030419 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:55.141274 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:55.141222 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:49:57.462266 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:49:57.462223 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:06.173525 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.173491 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:06.173963 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.173585 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vvd4b"] Jul 22 16:50:06.173963 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.173919 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd42cc2f-a202-4f27-abb5-9a95044001a9" containerName="manager" Jul 22 16:50:06.173963 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.173929 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd42cc2f-a202-4f27-abb5-9a95044001a9" containerName="manager" Jul 22 16:50:06.173963 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.173938 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40104b83-e12a-480c-95fc-569835ec64bb" containerName="manager" Jul 22 16:50:06.173963 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.173943 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="40104b83-e12a-480c-95fc-569835ec64bb" containerName="manager" Jul 22 16:50:06.174150 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.174001 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="40104b83-e12a-480c-95fc-569835ec64bb" containerName="manager" Jul 22 16:50:06.174150 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.174009 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd42cc2f-a202-4f27-abb5-9a95044001a9" containerName="manager" Jul 22 16:50:06.175858 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.175840 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" Jul 22 16:50:06.178623 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.178600 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bfjkn\"" Jul 22 16:50:06.178720 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.178636 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 16:50:06.179788 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.179772 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 16:50:06.179839 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.179774 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 16:50:06.184957 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.184935 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vvd4b"] Jul 22 16:50:06.201880 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.201848 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qdcg\" (UniqueName: \"kubernetes.io/projected/863cbe69-c641-41fe-8a0a-423d5bed4dc1-kube-api-access-7qdcg\") pod \"limitador-limitador-64c94fd64f-vvd4b\" (UID: \"863cbe69-c641-41fe-8a0a-423d5bed4dc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" Jul 22 16:50:06.202067 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.201955 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/863cbe69-c641-41fe-8a0a-423d5bed4dc1-config-file\") pod \"limitador-limitador-64c94fd64f-vvd4b\" (UID: \"863cbe69-c641-41fe-8a0a-423d5bed4dc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" Jul 22 16:50:06.271792 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.271753 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vvd4b"] Jul 22 16:50:06.282172 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.282133 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:06.302422 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.302369 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/863cbe69-c641-41fe-8a0a-423d5bed4dc1-config-file\") pod \"limitador-limitador-64c94fd64f-vvd4b\" (UID: \"863cbe69-c641-41fe-8a0a-423d5bed4dc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" Jul 22 16:50:06.302587 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.302448 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qdcg\" (UniqueName: \"kubernetes.io/projected/863cbe69-c641-41fe-8a0a-423d5bed4dc1-kube-api-access-7qdcg\") pod \"limitador-limitador-64c94fd64f-vvd4b\" (UID: \"863cbe69-c641-41fe-8a0a-423d5bed4dc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" Jul 22 16:50:06.302996 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.302974 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/863cbe69-c641-41fe-8a0a-423d5bed4dc1-config-file\") pod \"limitador-limitador-64c94fd64f-vvd4b\" (UID: \"863cbe69-c641-41fe-8a0a-423d5bed4dc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" Jul 22 16:50:06.310890 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.310862 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qdcg\" (UniqueName: \"kubernetes.io/projected/863cbe69-c641-41fe-8a0a-423d5bed4dc1-kube-api-access-7qdcg\") pod \"limitador-limitador-64c94fd64f-vvd4b\" (UID: \"863cbe69-c641-41fe-8a0a-423d5bed4dc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" Jul 22 16:50:06.381226 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.381174 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:06.489132 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.489098 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" Jul 22 16:50:06.620151 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:06.620120 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vvd4b"] Jul 22 16:50:06.622197 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:50:06.622160 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863cbe69_c641_41fe_8a0a_423d5bed4dc1.slice/crio-f24a1584c4d317813b938a5d795dc893896e2dee01b68da5ea4ce894c783b965 WatchSource:0}: Error finding container f24a1584c4d317813b938a5d795dc893896e2dee01b68da5ea4ce894c783b965: Status 404 returned error can't find the container with id f24a1584c4d317813b938a5d795dc893896e2dee01b68da5ea4ce894c783b965 Jul 22 16:50:07.012379 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:07.012346 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:07.076337 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:07.076307 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:07.237327 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:07.237275 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" event={"ID":"863cbe69-c641-41fe-8a0a-423d5bed4dc1","Type":"ContainerStarted","Data":"f24a1584c4d317813b938a5d795dc893896e2dee01b68da5ea4ce894c783b965"} Jul 22 16:50:10.253136 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:10.253103 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" event={"ID":"863cbe69-c641-41fe-8a0a-423d5bed4dc1","Type":"ContainerStarted","Data":"5fb82731fa11c4a38808f327e5c2f853325cbd8bd7dfbe68cabb2082314cc727"} Jul 22 16:50:10.253560 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:10.253216 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" Jul 22 16:50:10.271602 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:10.271549 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" podStartSLOduration=1.641695313 podStartE2EDuration="4.27153358s" totalImagesPullingTime="2.629838267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:50:10.270568527 +0000 UTC m=+575.247296844" watchObservedRunningTime="2026-07-22 16:50:10.27153358 +0000 UTC m=+575.248261896" Jul 22 16:50:21.259090 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:21.259057 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" Jul 22 16:50:21.512455 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:21.512373 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vvd4b"] Jul 22 16:50:21.512614 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:21.512591 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" podUID="863cbe69-c641-41fe-8a0a-423d5bed4dc1" containerName="limitador" containerID="cri-o://5fb82731fa11c4a38808f327e5c2f853325cbd8bd7dfbe68cabb2082314cc727" gracePeriod=30 Jul 22 16:50:22.055851 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.055823 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" Jul 22 16:50:22.132187 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.132107 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qdcg\" (UniqueName: \"kubernetes.io/projected/863cbe69-c641-41fe-8a0a-423d5bed4dc1-kube-api-access-7qdcg\") pod \"863cbe69-c641-41fe-8a0a-423d5bed4dc1\" (UID: \"863cbe69-c641-41fe-8a0a-423d5bed4dc1\") " Jul 22 16:50:22.132187 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.132155 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/863cbe69-c641-41fe-8a0a-423d5bed4dc1-config-file\") pod \"863cbe69-c641-41fe-8a0a-423d5bed4dc1\" (UID: \"863cbe69-c641-41fe-8a0a-423d5bed4dc1\") " Jul 22 16:50:22.132517 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.132496 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863cbe69-c641-41fe-8a0a-423d5bed4dc1-config-file" (OuterVolumeSpecName: "config-file") pod "863cbe69-c641-41fe-8a0a-423d5bed4dc1" (UID: "863cbe69-c641-41fe-8a0a-423d5bed4dc1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 16:50:22.134089 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.134064 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863cbe69-c641-41fe-8a0a-423d5bed4dc1-kube-api-access-7qdcg" (OuterVolumeSpecName: "kube-api-access-7qdcg") pod "863cbe69-c641-41fe-8a0a-423d5bed4dc1" (UID: "863cbe69-c641-41fe-8a0a-423d5bed4dc1"). InnerVolumeSpecName "kube-api-access-7qdcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:50:22.233231 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.233197 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qdcg\" (UniqueName: \"kubernetes.io/projected/863cbe69-c641-41fe-8a0a-423d5bed4dc1-kube-api-access-7qdcg\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:50:22.233231 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.233223 2557 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/863cbe69-c641-41fe-8a0a-423d5bed4dc1-config-file\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:50:22.309304 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.309271 2557 generic.go:356] "Generic (PLEG): container finished" podID="863cbe69-c641-41fe-8a0a-423d5bed4dc1" containerID="5fb82731fa11c4a38808f327e5c2f853325cbd8bd7dfbe68cabb2082314cc727" exitCode=0 Jul 22 16:50:22.309679 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.309316 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" Jul 22 16:50:22.309679 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.309359 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" event={"ID":"863cbe69-c641-41fe-8a0a-423d5bed4dc1","Type":"ContainerDied","Data":"5fb82731fa11c4a38808f327e5c2f853325cbd8bd7dfbe68cabb2082314cc727"} Jul 22 16:50:22.309679 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.309420 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vvd4b" event={"ID":"863cbe69-c641-41fe-8a0a-423d5bed4dc1","Type":"ContainerDied","Data":"f24a1584c4d317813b938a5d795dc893896e2dee01b68da5ea4ce894c783b965"} Jul 22 16:50:22.309679 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.309451 2557 scope.go:117] "RemoveContainer" containerID="5fb82731fa11c4a38808f327e5c2f853325cbd8bd7dfbe68cabb2082314cc727" Jul 22 16:50:22.319924 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.319908 2557 scope.go:117] "RemoveContainer" containerID="5fb82731fa11c4a38808f327e5c2f853325cbd8bd7dfbe68cabb2082314cc727" Jul 22 16:50:22.320197 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:50:22.320177 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb82731fa11c4a38808f327e5c2f853325cbd8bd7dfbe68cabb2082314cc727\": container with ID starting with 5fb82731fa11c4a38808f327e5c2f853325cbd8bd7dfbe68cabb2082314cc727 not found: ID does not exist" containerID="5fb82731fa11c4a38808f327e5c2f853325cbd8bd7dfbe68cabb2082314cc727" Jul 22 16:50:22.320273 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.320206 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb82731fa11c4a38808f327e5c2f853325cbd8bd7dfbe68cabb2082314cc727"} err="failed to get container status \"5fb82731fa11c4a38808f327e5c2f853325cbd8bd7dfbe68cabb2082314cc727\": rpc error: code = NotFound desc = could not find container \"5fb82731fa11c4a38808f327e5c2f853325cbd8bd7dfbe68cabb2082314cc727\": container with ID starting with 5fb82731fa11c4a38808f327e5c2f853325cbd8bd7dfbe68cabb2082314cc727 not found: ID does not exist" Jul 22 16:50:22.332105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.332077 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vvd4b"] Jul 22 16:50:22.337597 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.337572 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vvd4b"] Jul 22 16:50:22.339852 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.339828 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:22.399932 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:22.399863 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:23.574703 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:23.574672 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="863cbe69-c641-41fe-8a0a-423d5bed4dc1" path="/var/lib/kubelet/pods/863cbe69-c641-41fe-8a0a-423d5bed4dc1/volumes" Jul 22 16:50:23.905983 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:23.905900 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:24.006399 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.006365 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:24.337818 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.337786 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8"] Jul 22 16:50:24.338195 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.338180 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863cbe69-c641-41fe-8a0a-423d5bed4dc1" containerName="limitador" Jul 22 16:50:24.338252 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.338198 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="863cbe69-c641-41fe-8a0a-423d5bed4dc1" containerName="limitador" Jul 22 16:50:24.338288 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.338266 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="863cbe69-c641-41fe-8a0a-423d5bed4dc1" containerName="limitador" Jul 22 16:50:24.340472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.340454 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8" Jul 22 16:50:24.343220 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.343197 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 22 16:50:24.343355 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.343227 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 22 16:50:24.344346 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.344327 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 16:50:24.344445 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.344327 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-lk74d\"" Jul 22 16:50:24.348918 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.348893 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8"] Jul 22 16:50:24.372263 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.372233 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:24.451629 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.451597 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/667d8c5d-35c7-48e5-8800-70b3f22aa3bf-data\") pod \"postgres-69fb4594cf-4w9t8\" (UID: \"667d8c5d-35c7-48e5-8800-70b3f22aa3bf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8" Jul 22 16:50:24.451781 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.451669 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnc7c\" (UniqueName: \"kubernetes.io/projected/667d8c5d-35c7-48e5-8800-70b3f22aa3bf-kube-api-access-cnc7c\") pod \"postgres-69fb4594cf-4w9t8\" (UID: \"667d8c5d-35c7-48e5-8800-70b3f22aa3bf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8" Jul 22 16:50:24.552419 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.552384 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/667d8c5d-35c7-48e5-8800-70b3f22aa3bf-data\") pod \"postgres-69fb4594cf-4w9t8\" (UID: \"667d8c5d-35c7-48e5-8800-70b3f22aa3bf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8" Jul 22 16:50:24.552579 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.552440 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnc7c\" (UniqueName: \"kubernetes.io/projected/667d8c5d-35c7-48e5-8800-70b3f22aa3bf-kube-api-access-cnc7c\") pod \"postgres-69fb4594cf-4w9t8\" (UID: \"667d8c5d-35c7-48e5-8800-70b3f22aa3bf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8" Jul 22 16:50:24.552761 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.552742 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/667d8c5d-35c7-48e5-8800-70b3f22aa3bf-data\") pod \"postgres-69fb4594cf-4w9t8\" (UID: \"667d8c5d-35c7-48e5-8800-70b3f22aa3bf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8" Jul 22 16:50:24.561271 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.561241 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnc7c\" (UniqueName: \"kubernetes.io/projected/667d8c5d-35c7-48e5-8800-70b3f22aa3bf-kube-api-access-cnc7c\") pod \"postgres-69fb4594cf-4w9t8\" (UID: \"667d8c5d-35c7-48e5-8800-70b3f22aa3bf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8" Jul 22 16:50:24.650854 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.650780 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8" Jul 22 16:50:24.783981 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:24.783948 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8"] Jul 22 16:50:24.787072 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:50:24.787045 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667d8c5d_35c7_48e5_8800_70b3f22aa3bf.slice/crio-a6b7bb9899a78f0788a26bd7a12c0fb3c4996dc2249c6c287381511a868af5c3 WatchSource:0}: Error finding container a6b7bb9899a78f0788a26bd7a12c0fb3c4996dc2249c6c287381511a868af5c3: Status 404 returned error can't find the container with id a6b7bb9899a78f0788a26bd7a12c0fb3c4996dc2249c6c287381511a868af5c3 Jul 22 16:50:25.325105 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:25.325067 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8" event={"ID":"667d8c5d-35c7-48e5-8800-70b3f22aa3bf","Type":"ContainerStarted","Data":"a6b7bb9899a78f0788a26bd7a12c0fb3c4996dc2249c6c287381511a868af5c3"} Jul 22 16:50:30.351483 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:30.351445 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8" event={"ID":"667d8c5d-35c7-48e5-8800-70b3f22aa3bf","Type":"ContainerStarted","Data":"a059485336ccef83a382e5eb150a6e06e1d2f8e631817efe8248fcdb9d721006"} Jul 22 16:50:30.351969 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:30.351513 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8" Jul 22 16:50:30.367976 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:30.367909 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8" podStartSLOduration=1.5659879939999999 podStartE2EDuration="6.367891084s" totalImagesPullingTime="4.80190309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:50:30.367538004 +0000 UTC m=+595.344266322" watchObservedRunningTime="2026-07-22 16:50:30.367891084 +0000 UTC m=+595.344619401" Jul 22 16:50:30.716639 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:30.716558 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:36.392458 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:36.392430 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4w9t8" Jul 22 16:50:36.705712 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:36.705620 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:37.170247 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:37.170212 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:38.902802 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:38.902767 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:42.194201 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:42.194165 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:43.730241 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.730208 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-868fd4f75f-smgsl"] Jul 22 16:50:43.734301 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.734275 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:43.746676 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.746651 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-868fd4f75f-smgsl"] Jul 22 16:50:43.746808 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.746743 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-868fd4f75f-smgsl" Jul 22 16:50:43.749467 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.749444 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wmd78\"" Jul 22 16:50:43.749606 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.749445 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 16:50:43.749606 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.749449 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 16:50:43.826626 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.826596 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-586sd\" (UniqueName: \"kubernetes.io/projected/71411eeb-77ce-4ca8-a625-b068f60badc5-kube-api-access-586sd\") pod \"maas-controller-868fd4f75f-smgsl\" (UID: \"71411eeb-77ce-4ca8-a625-b068f60badc5\") " pod="opendatahub/maas-controller-868fd4f75f-smgsl" Jul 22 16:50:43.826798 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.826652 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71411eeb-77ce-4ca8-a625-b068f60badc5-webhook-cert\") pod \"maas-controller-868fd4f75f-smgsl\" (UID: \"71411eeb-77ce-4ca8-a625-b068f60badc5\") " pod="opendatahub/maas-controller-868fd4f75f-smgsl" Jul 22 16:50:43.833445 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.833420 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:43.921579 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.921541 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:43.922512 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.922486 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58867cc484-f7fqt"] Jul 22 16:50:43.925963 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.925943 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58867cc484-f7fqt" Jul 22 16:50:43.927075 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.927055 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-586sd\" (UniqueName: \"kubernetes.io/projected/71411eeb-77ce-4ca8-a625-b068f60badc5-kube-api-access-586sd\") pod \"maas-controller-868fd4f75f-smgsl\" (UID: \"71411eeb-77ce-4ca8-a625-b068f60badc5\") " pod="opendatahub/maas-controller-868fd4f75f-smgsl" Jul 22 16:50:43.927155 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.927097 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71411eeb-77ce-4ca8-a625-b068f60badc5-webhook-cert\") pod \"maas-controller-868fd4f75f-smgsl\" (UID: \"71411eeb-77ce-4ca8-a625-b068f60badc5\") " pod="opendatahub/maas-controller-868fd4f75f-smgsl" Jul 22 16:50:43.929458 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.929441 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71411eeb-77ce-4ca8-a625-b068f60badc5-webhook-cert\") pod \"maas-controller-868fd4f75f-smgsl\" (UID: \"71411eeb-77ce-4ca8-a625-b068f60badc5\") " pod="opendatahub/maas-controller-868fd4f75f-smgsl" Jul 22 16:50:43.936346 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.936276 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58867cc484-f7fqt"] Jul 22 16:50:43.938344 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:43.938189 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-586sd\" (UniqueName: \"kubernetes.io/projected/71411eeb-77ce-4ca8-a625-b068f60badc5-kube-api-access-586sd\") pod \"maas-controller-868fd4f75f-smgsl\" (UID: \"71411eeb-77ce-4ca8-a625-b068f60badc5\") " pod="opendatahub/maas-controller-868fd4f75f-smgsl" Jul 22 16:50:44.028592 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:44.028510 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8391846c-50c7-46a0-9111-187e003f836f-webhook-cert\") pod \"maas-controller-58867cc484-f7fqt\" (UID: \"8391846c-50c7-46a0-9111-187e003f836f\") " pod="opendatahub/maas-controller-58867cc484-f7fqt" Jul 22 16:50:44.028739 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:44.028611 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcct8\" (UniqueName: \"kubernetes.io/projected/8391846c-50c7-46a0-9111-187e003f836f-kube-api-access-xcct8\") pod \"maas-controller-58867cc484-f7fqt\" (UID: \"8391846c-50c7-46a0-9111-187e003f836f\") " pod="opendatahub/maas-controller-58867cc484-f7fqt" Jul 22 16:50:44.057482 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:44.057431 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-868fd4f75f-smgsl" Jul 22 16:50:44.129764 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:44.129732 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcct8\" (UniqueName: \"kubernetes.io/projected/8391846c-50c7-46a0-9111-187e003f836f-kube-api-access-xcct8\") pod \"maas-controller-58867cc484-f7fqt\" (UID: \"8391846c-50c7-46a0-9111-187e003f836f\") " pod="opendatahub/maas-controller-58867cc484-f7fqt" Jul 22 16:50:44.129914 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:44.129789 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8391846c-50c7-46a0-9111-187e003f836f-webhook-cert\") pod \"maas-controller-58867cc484-f7fqt\" (UID: \"8391846c-50c7-46a0-9111-187e003f836f\") " pod="opendatahub/maas-controller-58867cc484-f7fqt" Jul 22 16:50:44.132308 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:44.132284 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8391846c-50c7-46a0-9111-187e003f836f-webhook-cert\") pod \"maas-controller-58867cc484-f7fqt\" (UID: \"8391846c-50c7-46a0-9111-187e003f836f\") " pod="opendatahub/maas-controller-58867cc484-f7fqt" Jul 22 16:50:44.139197 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:44.139157 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcct8\" (UniqueName: \"kubernetes.io/projected/8391846c-50c7-46a0-9111-187e003f836f-kube-api-access-xcct8\") pod \"maas-controller-58867cc484-f7fqt\" (UID: \"8391846c-50c7-46a0-9111-187e003f836f\") " pod="opendatahub/maas-controller-58867cc484-f7fqt" Jul 22 16:50:44.252879 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:44.252848 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58867cc484-f7fqt" Jul 22 16:50:44.380909 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:44.380879 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58867cc484-f7fqt"] Jul 22 16:50:44.382737 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:50:44.382707 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8391846c_50c7_46a0_9111_187e003f836f.slice/crio-3c8528be16d5474ac67d2bf019c25da7908ef42706b021fa019fb76284708b28 WatchSource:0}: Error finding container 3c8528be16d5474ac67d2bf019c25da7908ef42706b021fa019fb76284708b28: Status 404 returned error can't find the container with id 3c8528be16d5474ac67d2bf019c25da7908ef42706b021fa019fb76284708b28 Jul 22 16:50:44.394736 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:44.394714 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-868fd4f75f-smgsl"] Jul 22 16:50:44.396368 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:50:44.396339 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71411eeb_77ce_4ca8_a625_b068f60badc5.slice/crio-17cd1d943d0c26056ef029ddc266f3828880137dff563ef2cb98261b21a6643b WatchSource:0}: Error finding container 17cd1d943d0c26056ef029ddc266f3828880137dff563ef2cb98261b21a6643b: Status 404 returned error can't find the container with id 17cd1d943d0c26056ef029ddc266f3828880137dff563ef2cb98261b21a6643b Jul 22 16:50:44.422828 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:44.422792 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58867cc484-f7fqt" event={"ID":"8391846c-50c7-46a0-9111-187e003f836f","Type":"ContainerStarted","Data":"3c8528be16d5474ac67d2bf019c25da7908ef42706b021fa019fb76284708b28"} Jul 22 16:50:44.424051 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:44.424011 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-868fd4f75f-smgsl" event={"ID":"71411eeb-77ce-4ca8-a625-b068f60badc5","Type":"ContainerStarted","Data":"17cd1d943d0c26056ef029ddc266f3828880137dff563ef2cb98261b21a6643b"} Jul 22 16:50:47.132448 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:47.132412 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:47.232081 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:47.232049 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:47.444834 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:47.444792 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58867cc484-f7fqt" event={"ID":"8391846c-50c7-46a0-9111-187e003f836f","Type":"ContainerStarted","Data":"7603e8bb02ad60da8dc9c3ecc0960b9b640d0fc9beb0db6cbbb12e6a984f5f90"} Jul 22 16:50:47.445053 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:47.444901 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58867cc484-f7fqt" Jul 22 16:50:47.446359 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:47.446336 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-868fd4f75f-smgsl" event={"ID":"71411eeb-77ce-4ca8-a625-b068f60badc5","Type":"ContainerStarted","Data":"c96c966254999f5b4f469daf361862c3c2af48ba8f9e28005dad90376ad9a6c4"} Jul 22 16:50:47.446467 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:47.446388 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-868fd4f75f-smgsl" Jul 22 16:50:47.467600 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:47.467555 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58867cc484-f7fqt" podStartSLOduration=1.919385068 podStartE2EDuration="4.467542045s" totalImagesPullingTime="2.548156977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:50:47.46520862 +0000 UTC m=+612.441936936" watchObservedRunningTime="2026-07-22 16:50:47.467542045 +0000 UTC m=+612.444270402" Jul 22 16:50:47.490804 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:47.490714 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-868fd4f75f-smgsl" podStartSLOduration=1.946184073 podStartE2EDuration="4.490700743s" totalImagesPullingTime="2.54451667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:50:47.488823543 +0000 UTC m=+612.465551859" watchObservedRunningTime="2026-07-22 16:50:47.490700743 +0000 UTC m=+612.467429059" Jul 22 16:50:50.067356 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:50.067318 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:50:58.454302 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:58.454227 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-868fd4f75f-smgsl" Jul 22 16:50:58.454302 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:58.454302 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58867cc484-f7fqt" Jul 22 16:50:58.545943 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:58.545904 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-868fd4f75f-smgsl"] Jul 22 16:50:58.546210 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:58.546161 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-868fd4f75f-smgsl" podUID="71411eeb-77ce-4ca8-a625-b068f60badc5" containerName="manager" containerID="cri-o://c96c966254999f5b4f469daf361862c3c2af48ba8f9e28005dad90376ad9a6c4" gracePeriod=30 Jul 22 16:50:58.786978 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:58.786954 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-868fd4f75f-smgsl" Jul 22 16:50:58.862755 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:58.862723 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-586sd\" (UniqueName: \"kubernetes.io/projected/71411eeb-77ce-4ca8-a625-b068f60badc5-kube-api-access-586sd\") pod \"71411eeb-77ce-4ca8-a625-b068f60badc5\" (UID: \"71411eeb-77ce-4ca8-a625-b068f60badc5\") " Jul 22 16:50:58.862939 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:58.862921 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71411eeb-77ce-4ca8-a625-b068f60badc5-webhook-cert\") pod \"71411eeb-77ce-4ca8-a625-b068f60badc5\" (UID: \"71411eeb-77ce-4ca8-a625-b068f60badc5\") " Jul 22 16:50:58.864889 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:58.864852 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71411eeb-77ce-4ca8-a625-b068f60badc5-kube-api-access-586sd" (OuterVolumeSpecName: "kube-api-access-586sd") pod "71411eeb-77ce-4ca8-a625-b068f60badc5" (UID: "71411eeb-77ce-4ca8-a625-b068f60badc5"). InnerVolumeSpecName "kube-api-access-586sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:50:58.864998 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:58.864917 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71411eeb-77ce-4ca8-a625-b068f60badc5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "71411eeb-77ce-4ca8-a625-b068f60badc5" (UID: "71411eeb-77ce-4ca8-a625-b068f60badc5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:50:58.964416 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:58.964335 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-586sd\" (UniqueName: \"kubernetes.io/projected/71411eeb-77ce-4ca8-a625-b068f60badc5-kube-api-access-586sd\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:50:58.964416 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:58.964364 2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/71411eeb-77ce-4ca8-a625-b068f60badc5-webhook-cert\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:50:59.505228 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:59.505196 2557 generic.go:356] "Generic (PLEG): container finished" podID="71411eeb-77ce-4ca8-a625-b068f60badc5" containerID="c96c966254999f5b4f469daf361862c3c2af48ba8f9e28005dad90376ad9a6c4" exitCode=0 Jul 22 16:50:59.505734 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:59.505255 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-868fd4f75f-smgsl" Jul 22 16:50:59.505734 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:59.505267 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-868fd4f75f-smgsl" event={"ID":"71411eeb-77ce-4ca8-a625-b068f60badc5","Type":"ContainerDied","Data":"c96c966254999f5b4f469daf361862c3c2af48ba8f9e28005dad90376ad9a6c4"} Jul 22 16:50:59.505734 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:59.505312 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-868fd4f75f-smgsl" event={"ID":"71411eeb-77ce-4ca8-a625-b068f60badc5","Type":"ContainerDied","Data":"17cd1d943d0c26056ef029ddc266f3828880137dff563ef2cb98261b21a6643b"} Jul 22 16:50:59.505734 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:59.505336 2557 scope.go:117] "RemoveContainer" containerID="c96c966254999f5b4f469daf361862c3c2af48ba8f9e28005dad90376ad9a6c4" Jul 22 16:50:59.514556 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:59.514540 2557 scope.go:117] "RemoveContainer" containerID="c96c966254999f5b4f469daf361862c3c2af48ba8f9e28005dad90376ad9a6c4" Jul 22 16:50:59.514804 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:50:59.514788 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c96c966254999f5b4f469daf361862c3c2af48ba8f9e28005dad90376ad9a6c4\": container with ID starting with c96c966254999f5b4f469daf361862c3c2af48ba8f9e28005dad90376ad9a6c4 not found: ID does not exist" containerID="c96c966254999f5b4f469daf361862c3c2af48ba8f9e28005dad90376ad9a6c4" Jul 22 16:50:59.514842 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:59.514812 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c96c966254999f5b4f469daf361862c3c2af48ba8f9e28005dad90376ad9a6c4"} err="failed to get container status \"c96c966254999f5b4f469daf361862c3c2af48ba8f9e28005dad90376ad9a6c4\": rpc error: code = NotFound desc = could not find container \"c96c966254999f5b4f469daf361862c3c2af48ba8f9e28005dad90376ad9a6c4\": container with ID starting with c96c966254999f5b4f469daf361862c3c2af48ba8f9e28005dad90376ad9a6c4 not found: ID does not exist" Jul 22 16:50:59.528233 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:59.528202 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-868fd4f75f-smgsl"] Jul 22 16:50:59.532343 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:59.532319 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-868fd4f75f-smgsl"] Jul 22 16:50:59.574525 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:50:59.574494 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71411eeb-77ce-4ca8-a625-b068f60badc5" path="/var/lib/kubelet/pods/71411eeb-77ce-4ca8-a625-b068f60badc5/volumes" Jul 22 16:51:19.181645 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.181604 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:51:19.280822 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.280785 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:51:19.749136 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.749099 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:51:19.794959 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.794926 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65996798d9-wb9dl"] Jul 22 16:51:19.795334 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.795321 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71411eeb-77ce-4ca8-a625-b068f60badc5" containerName="manager" Jul 22 16:51:19.795377 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.795336 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="71411eeb-77ce-4ca8-a625-b068f60badc5" containerName="manager" Jul 22 16:51:19.795413 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.795390 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="71411eeb-77ce-4ca8-a625-b068f60badc5" containerName="manager" Jul 22 16:51:19.799333 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.799314 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" Jul 22 16:51:19.801964 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.801941 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 22 16:51:19.811397 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.811367 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65996798d9-wb9dl"] Jul 22 16:51:19.840237 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.840212 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz6hf\" (UniqueName: \"kubernetes.io/projected/a6258786-d895-4d76-84da-b921ae0b611c-kube-api-access-lz6hf\") pod \"maas-api-65996798d9-wb9dl\" (UID: \"a6258786-d895-4d76-84da-b921ae0b611c\") " pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" Jul 22 16:51:19.840364 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.840273 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6258786-d895-4d76-84da-b921ae0b611c-maas-api-tls\") pod \"maas-api-65996798d9-wb9dl\" (UID: \"a6258786-d895-4d76-84da-b921ae0b611c\") " pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" Jul 22 16:51:19.840406 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.840361 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6258786-d895-4d76-84da-b921ae0b611c-openshift-service-ca\") pod \"maas-api-65996798d9-wb9dl\" (UID: \"a6258786-d895-4d76-84da-b921ae0b611c\") " pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" Jul 22 16:51:19.886518 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.886484 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:51:19.941936 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.941898 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6258786-d895-4d76-84da-b921ae0b611c-openshift-service-ca\") pod \"maas-api-65996798d9-wb9dl\" (UID: \"a6258786-d895-4d76-84da-b921ae0b611c\") " pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" Jul 22 16:51:19.942096 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.941977 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz6hf\" (UniqueName: \"kubernetes.io/projected/a6258786-d895-4d76-84da-b921ae0b611c-kube-api-access-lz6hf\") pod \"maas-api-65996798d9-wb9dl\" (UID: \"a6258786-d895-4d76-84da-b921ae0b611c\") " pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" Jul 22 16:51:19.942096 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.942067 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6258786-d895-4d76-84da-b921ae0b611c-maas-api-tls\") pod \"maas-api-65996798d9-wb9dl\" (UID: \"a6258786-d895-4d76-84da-b921ae0b611c\") " pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" Jul 22 16:51:19.942796 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.942764 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6258786-d895-4d76-84da-b921ae0b611c-openshift-service-ca\") pod \"maas-api-65996798d9-wb9dl\" (UID: \"a6258786-d895-4d76-84da-b921ae0b611c\") " pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" Jul 22 16:51:19.944824 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.944801 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6258786-d895-4d76-84da-b921ae0b611c-maas-api-tls\") pod \"maas-api-65996798d9-wb9dl\" (UID: \"a6258786-d895-4d76-84da-b921ae0b611c\") " pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" Jul 22 16:51:19.954150 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:19.954126 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz6hf\" (UniqueName: \"kubernetes.io/projected/a6258786-d895-4d76-84da-b921ae0b611c-kube-api-access-lz6hf\") pod \"maas-api-65996798d9-wb9dl\" (UID: \"a6258786-d895-4d76-84da-b921ae0b611c\") " pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" Jul 22 16:51:20.109639 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.109550 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" Jul 22 16:51:20.245451 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.245420 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65996798d9-wb9dl"] Jul 22 16:51:20.248848 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:51:20.248824 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6258786_d895_4d76_84da_b921ae0b611c.slice/crio-479370f9dc87c08945a25733daef67ddf01a504eabbf9f9c7d64ad75cbd82080 WatchSource:0}: Error finding container 479370f9dc87c08945a25733daef67ddf01a504eabbf9f9c7d64ad75cbd82080: Status 404 returned error can't find the container with id 479370f9dc87c08945a25733daef67ddf01a504eabbf9f9c7d64ad75cbd82080 Jul 22 16:51:20.268529 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.268498 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:51:20.363944 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.363857 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:51:20.368505 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.368479 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-vg25t"] Jul 22 16:51:20.371133 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.371112 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-vg25t" Jul 22 16:51:20.374137 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.374119 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 22 16:51:20.380361 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.380336 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-vg25t"] Jul 22 16:51:20.396829 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.396806 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-fp786"] Jul 22 16:51:20.399088 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.399069 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-fp786" Jul 22 16:51:20.401498 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.401478 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-n24xw\"" Jul 22 16:51:20.410253 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.410234 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-fp786"] Jul 22 16:51:20.445561 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.445534 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stk4l\" (UniqueName: \"kubernetes.io/projected/02b632be-5f97-401e-abc1-036f2033c085-kube-api-access-stk4l\") pod \"payload-pre-processing-7677b774bb-vg25t\" (UID: \"02b632be-5f97-401e-abc1-036f2033c085\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-vg25t" Jul 22 16:51:20.445674 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.445574 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2h4p\" (UniqueName: \"kubernetes.io/projected/eab14db0-c780-4ed8-9b19-93ee88b16537-kube-api-access-c2h4p\") pod \"payload-processing-5bf47cd988-fp786\" (UID: \"eab14db0-c780-4ed8-9b19-93ee88b16537\") " pod="openshift-ingress/payload-processing-5bf47cd988-fp786" Jul 22 16:51:20.445674 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.445613 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02b632be-5f97-401e-abc1-036f2033c085-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-vg25t\" (UID: \"02b632be-5f97-401e-abc1-036f2033c085\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-vg25t" Jul 22 16:51:20.445761 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.445694 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab14db0-c780-4ed8-9b19-93ee88b16537-plugins-config-volume\") pod \"payload-processing-5bf47cd988-fp786\" (UID: \"eab14db0-c780-4ed8-9b19-93ee88b16537\") " pod="openshift-ingress/payload-processing-5bf47cd988-fp786" Jul 22 16:51:20.546381 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.546350 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2h4p\" (UniqueName: \"kubernetes.io/projected/eab14db0-c780-4ed8-9b19-93ee88b16537-kube-api-access-c2h4p\") pod \"payload-processing-5bf47cd988-fp786\" (UID: \"eab14db0-c780-4ed8-9b19-93ee88b16537\") " pod="openshift-ingress/payload-processing-5bf47cd988-fp786" Jul 22 16:51:20.546546 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.546406 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02b632be-5f97-401e-abc1-036f2033c085-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-vg25t\" (UID: \"02b632be-5f97-401e-abc1-036f2033c085\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-vg25t" Jul 22 16:51:20.546546 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.546444 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab14db0-c780-4ed8-9b19-93ee88b16537-plugins-config-volume\") pod \"payload-processing-5bf47cd988-fp786\" (UID: \"eab14db0-c780-4ed8-9b19-93ee88b16537\") " pod="openshift-ingress/payload-processing-5bf47cd988-fp786" Jul 22 16:51:20.546546 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.546478 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stk4l\" (UniqueName: \"kubernetes.io/projected/02b632be-5f97-401e-abc1-036f2033c085-kube-api-access-stk4l\") pod \"payload-pre-processing-7677b774bb-vg25t\" (UID: \"02b632be-5f97-401e-abc1-036f2033c085\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-vg25t" Jul 22 16:51:20.547226 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.547207 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab14db0-c780-4ed8-9b19-93ee88b16537-plugins-config-volume\") pod \"payload-processing-5bf47cd988-fp786\" (UID: \"eab14db0-c780-4ed8-9b19-93ee88b16537\") " pod="openshift-ingress/payload-processing-5bf47cd988-fp786" Jul 22 16:51:20.547510 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.547485 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02b632be-5f97-401e-abc1-036f2033c085-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-vg25t\" (UID: \"02b632be-5f97-401e-abc1-036f2033c085\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-vg25t" Jul 22 16:51:20.556444 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.556415 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stk4l\" (UniqueName: \"kubernetes.io/projected/02b632be-5f97-401e-abc1-036f2033c085-kube-api-access-stk4l\") pod \"payload-pre-processing-7677b774bb-vg25t\" (UID: \"02b632be-5f97-401e-abc1-036f2033c085\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-vg25t" Jul 22 16:51:20.556560 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.556538 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2h4p\" (UniqueName: \"kubernetes.io/projected/eab14db0-c780-4ed8-9b19-93ee88b16537-kube-api-access-c2h4p\") pod \"payload-processing-5bf47cd988-fp786\" (UID: \"eab14db0-c780-4ed8-9b19-93ee88b16537\") " pod="openshift-ingress/payload-processing-5bf47cd988-fp786" Jul 22 16:51:20.599616 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.599574 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" event={"ID":"a6258786-d895-4d76-84da-b921ae0b611c","Type":"ContainerStarted","Data":"479370f9dc87c08945a25733daef67ddf01a504eabbf9f9c7d64ad75cbd82080"} Jul 22 16:51:20.683286 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.683199 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-vg25t" Jul 22 16:51:20.710114 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.710083 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-fp786" Jul 22 16:51:20.835148 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.835046 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-vg25t"] Jul 22 16:51:20.837649 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:51:20.837615 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b632be_5f97_401e_abc1_036f2033c085.slice/crio-d227c6a5341487098be5b1ce18c37a9d26e4493e3c60d5a0882605ebc832e354 WatchSource:0}: Error finding container d227c6a5341487098be5b1ce18c37a9d26e4493e3c60d5a0882605ebc832e354: Status 404 returned error can't find the container with id d227c6a5341487098be5b1ce18c37a9d26e4493e3c60d5a0882605ebc832e354 Jul 22 16:51:20.876107 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:20.876080 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-fp786"] Jul 22 16:51:20.878418 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:51:20.878390 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab14db0_c780_4ed8_9b19_93ee88b16537.slice/crio-3dab83e2957a45a284c239c170f02d5d528ff7987b3ce7cef504feb8f8494e4d WatchSource:0}: Error finding container 3dab83e2957a45a284c239c170f02d5d528ff7987b3ce7cef504feb8f8494e4d: Status 404 returned error can't find the container with id 3dab83e2957a45a284c239c170f02d5d528ff7987b3ce7cef504feb8f8494e4d Jul 22 16:51:21.610971 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:21.610884 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-vg25t" event={"ID":"02b632be-5f97-401e-abc1-036f2033c085","Type":"ContainerStarted","Data":"d227c6a5341487098be5b1ce18c37a9d26e4493e3c60d5a0882605ebc832e354"} Jul 22 16:51:21.613104 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:21.613038 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-fp786" event={"ID":"eab14db0-c780-4ed8-9b19-93ee88b16537","Type":"ContainerStarted","Data":"3dab83e2957a45a284c239c170f02d5d528ff7987b3ce7cef504feb8f8494e4d"} Jul 22 16:51:22.619719 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:22.619676 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" event={"ID":"a6258786-d895-4d76-84da-b921ae0b611c","Type":"ContainerStarted","Data":"aef3ead6231ab8efc5d7108b125a5fe2dcb1c77401b03cd55222970dc8852287"} Jul 22 16:51:22.620196 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:22.619789 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" Jul 22 16:51:22.639674 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:22.639622 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" podStartSLOduration=1.8776358229999999 podStartE2EDuration="3.639601857s" totalImagesPullingTime="1.761966034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:51:22.638084881 +0000 UTC m=+647.614813197" watchObservedRunningTime="2026-07-22 16:51:22.639601857 +0000 UTC m=+647.616330189" Jul 22 16:51:23.625649 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:23.625607 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-vg25t" event={"ID":"02b632be-5f97-401e-abc1-036f2033c085","Type":"ContainerStarted","Data":"71cc5afe56820bbd4503c53e986f82237adfa793a0b600777c07c9183a24974d"} Jul 22 16:51:23.626152 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:23.625706 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-vg25t" Jul 22 16:51:23.627207 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:23.627169 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-fp786" event={"ID":"eab14db0-c780-4ed8-9b19-93ee88b16537","Type":"ContainerStarted","Data":"2ebf391281706fdbc20aeb5583afb14db2bdbbf5ffec3e644d0f7df126c010cc"} Jul 22 16:51:23.627566 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:23.627343 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-fp786" Jul 22 16:51:23.643581 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:23.643534 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-vg25t" podStartSLOduration=1.695173155 podStartE2EDuration="3.643521105s" totalImagesPullingTime="1.94834795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:51:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:51:23.641637172 +0000 UTC m=+648.618365511" watchObservedRunningTime="2026-07-22 16:51:23.643521105 +0000 UTC m=+648.620249421" Jul 22 16:51:23.659537 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:23.659487 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-fp786" podStartSLOduration=1.758472463 podStartE2EDuration="3.65947433s" totalImagesPullingTime="1.901001867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:51:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:51:23.65693121 +0000 UTC m=+648.633659543" watchObservedRunningTime="2026-07-22 16:51:23.65947433 +0000 UTC m=+648.636202646" Jul 22 16:51:28.632562 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:28.632530 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65996798d9-wb9dl" Jul 22 16:51:30.493150 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:30.493092 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:51:30.592096 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:30.592056 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:51:34.633582 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:34.633544 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-vg25t" Jul 22 16:51:34.634051 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:34.633738 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-fp786" Jul 22 16:51:55.757472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:55.757440 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:51:55.809607 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:55.809573 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:51:57.510689 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.510647 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l"] Jul 22 16:51:57.513935 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.513907 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.517994 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.517970 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 16:51:57.518146 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.517973 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 16:51:57.518146 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.518012 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7d4dj\"" Jul 22 16:51:57.518146 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.517973 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 22 16:51:57.524046 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.524002 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l"] Jul 22 16:51:57.587747 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.587701 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/162d131d-1c24-471b-8332-1e20c9e1b701-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.587747 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.587741 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/162d131d-1c24-471b-8332-1e20c9e1b701-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.587988 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.587869 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qcmj\" (UniqueName: \"kubernetes.io/projected/162d131d-1c24-471b-8332-1e20c9e1b701-kube-api-access-5qcmj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.587988 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.587941 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/162d131d-1c24-471b-8332-1e20c9e1b701-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.588141 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.588018 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162d131d-1c24-471b-8332-1e20c9e1b701-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.588197 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.588168 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/162d131d-1c24-471b-8332-1e20c9e1b701-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.589560 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.589537 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:51:57.689622 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.689589 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162d131d-1c24-471b-8332-1e20c9e1b701-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.689790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.689639 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/162d131d-1c24-471b-8332-1e20c9e1b701-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.689790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.689673 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/162d131d-1c24-471b-8332-1e20c9e1b701-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.689790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.689691 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/162d131d-1c24-471b-8332-1e20c9e1b701-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.689790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.689729 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qcmj\" (UniqueName: \"kubernetes.io/projected/162d131d-1c24-471b-8332-1e20c9e1b701-kube-api-access-5qcmj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.689790 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.689760 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/162d131d-1c24-471b-8332-1e20c9e1b701-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.690181 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.690153 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/162d131d-1c24-471b-8332-1e20c9e1b701-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.690273 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.690184 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/162d131d-1c24-471b-8332-1e20c9e1b701-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.690273 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.690203 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/162d131d-1c24-471b-8332-1e20c9e1b701-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.691986 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.691970 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/162d131d-1c24-471b-8332-1e20c9e1b701-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.692272 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.692254 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/162d131d-1c24-471b-8332-1e20c9e1b701-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.698593 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.698569 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qcmj\" (UniqueName: \"kubernetes.io/projected/162d131d-1c24-471b-8332-1e20c9e1b701-kube-api-access-5qcmj\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l\" (UID: \"162d131d-1c24-471b-8332-1e20c9e1b701\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.825745 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.825665 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:51:57.955056 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.955013 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l"] Jul 22 16:51:57.957570 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:51:57.957539 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162d131d_1c24_471b_8332_1e20c9e1b701.slice/crio-2aa044ef7693d3baf1413c9e2352daa9716b6a6cf693f9f31138d9de5383ea71 WatchSource:0}: Error finding container 2aa044ef7693d3baf1413c9e2352daa9716b6a6cf693f9f31138d9de5383ea71: Status 404 returned error can't find the container with id 2aa044ef7693d3baf1413c9e2352daa9716b6a6cf693f9f31138d9de5383ea71 Jul 22 16:51:57.958847 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:57.958826 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 16:51:58.794182 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:58.794148 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" event={"ID":"162d131d-1c24-471b-8332-1e20c9e1b701","Type":"ContainerStarted","Data":"2aa044ef7693d3baf1413c9e2352daa9716b6a6cf693f9f31138d9de5383ea71"} Jul 22 16:51:59.800736 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:59.800699 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" event={"ID":"162d131d-1c24-471b-8332-1e20c9e1b701","Type":"ContainerStarted","Data":"1715922792cba248bbebb9af5bcc1117ad1a40bfbfbe201e6050ad60c66037b2"} Jul 22 16:51:59.821556 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:51:59.821494 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" podStartSLOduration=1.840300462 podStartE2EDuration="2.82147774s" totalImagesPullingTime="981.177278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:51:59.820575282 +0000 UTC m=+684.797303593" watchObservedRunningTime="2026-07-22 16:51:59.82147774 +0000 UTC m=+684.798206055" Jul 22 16:52:07.826140 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:07.826107 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:52:07.826612 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:07.826201 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:52:07.838982 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:07.838953 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:52:07.851228 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:07.851197 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l" Jul 22 16:52:12.233799 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:12.233765 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:52:19.978842 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:19.978805 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj"] Jul 22 16:52:19.983219 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:19.983195 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:19.985930 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:19.985908 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 22 16:52:19.991540 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:19.991514 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj"] Jul 22 16:52:20.027000 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.026962 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:52:20.087128 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.087095 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea69359-b394-4b9c-8b44-fec16e5df47b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.087309 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.087136 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea69359-b394-4b9c-8b44-fec16e5df47b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.087309 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.087165 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea69359-b394-4b9c-8b44-fec16e5df47b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.087309 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.087237 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl7s6\" (UniqueName: \"kubernetes.io/projected/fea69359-b394-4b9c-8b44-fec16e5df47b-kube-api-access-rl7s6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.087309 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.087288 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea69359-b394-4b9c-8b44-fec16e5df47b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.087492 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.087364 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea69359-b394-4b9c-8b44-fec16e5df47b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.187999 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.187969 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea69359-b394-4b9c-8b44-fec16e5df47b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.187999 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.188002 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea69359-b394-4b9c-8b44-fec16e5df47b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.188226 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.188020 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl7s6\" (UniqueName: \"kubernetes.io/projected/fea69359-b394-4b9c-8b44-fec16e5df47b-kube-api-access-rl7s6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.188226 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.188073 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea69359-b394-4b9c-8b44-fec16e5df47b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.188226 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.188106 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea69359-b394-4b9c-8b44-fec16e5df47b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.188226 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.188163 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea69359-b394-4b9c-8b44-fec16e5df47b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.188433 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.188409 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea69359-b394-4b9c-8b44-fec16e5df47b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.188582 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.188561 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea69359-b394-4b9c-8b44-fec16e5df47b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.188647 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.188588 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea69359-b394-4b9c-8b44-fec16e5df47b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.190379 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.190353 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea69359-b394-4b9c-8b44-fec16e5df47b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.190551 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.190534 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea69359-b394-4b9c-8b44-fec16e5df47b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.196648 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.196628 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl7s6\" (UniqueName: \"kubernetes.io/projected/fea69359-b394-4b9c-8b44-fec16e5df47b-kube-api-access-rl7s6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lz4qj\" (UID: \"fea69359-b394-4b9c-8b44-fec16e5df47b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.296165 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.296085 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:20.425121 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.425094 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj"] Jul 22 16:52:20.427399 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:52:20.427374 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea69359_b394_4b9c_8b44_fec16e5df47b.slice/crio-1317f115d40fa716cf5139712238555db40c00dac467de9302b1e14089f6e191 WatchSource:0}: Error finding container 1317f115d40fa716cf5139712238555db40c00dac467de9302b1e14089f6e191: Status 404 returned error can't find the container with id 1317f115d40fa716cf5139712238555db40c00dac467de9302b1e14089f6e191 Jul 22 16:52:20.904612 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.904572 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" event={"ID":"fea69359-b394-4b9c-8b44-fec16e5df47b","Type":"ContainerStarted","Data":"5c72066bd01c2c9065d34c90c1835f7524c68a5477f583d8d106655a3af1f253"} Jul 22 16:52:20.904808 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.904625 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" event={"ID":"fea69359-b394-4b9c-8b44-fec16e5df47b","Type":"ContainerStarted","Data":"1317f115d40fa716cf5139712238555db40c00dac467de9302b1e14089f6e191"} Jul 22 16:52:20.924239 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:20.924197 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" podStartSLOduration=1.6781506959999999 podStartE2EDuration="1.924180409s" totalImagesPullingTime="246.029713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:52:20.92360783 +0000 UTC m=+705.900336153" watchObservedRunningTime="2026-07-22 16:52:20.924180409 +0000 UTC m=+705.900908725" Jul 22 16:52:23.231319 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:23.231271 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:52:23.333653 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:23.333615 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:52:24.316710 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:24.316669 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:52:30.296366 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:30.296286 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:30.296366 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:30.296350 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:30.309449 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:30.309417 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:30.962099 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:30.962071 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lz4qj" Jul 22 16:52:32.533663 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:32.533626 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:52:45.021792 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:52:45.021752 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:53:37.361300 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:53:37.361263 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:53:37.456807 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:53:37.456768 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:53:38.507224 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:53:38.507170 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:53:38.639344 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:53:38.639292 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:54:28.948111 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:28.948072 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58867cc484-f7fqt"] Jul 22 16:54:28.948727 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:28.948327 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58867cc484-f7fqt" podUID="8391846c-50c7-46a0-9111-187e003f836f" containerName="manager" containerID="cri-o://7603e8bb02ad60da8dc9c3ecc0960b9b640d0fc9beb0db6cbbb12e6a984f5f90" gracePeriod=30 Jul 22 16:54:29.200826 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.200773 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58867cc484-f7fqt" Jul 22 16:54:29.265507 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.265483 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8391846c-50c7-46a0-9111-187e003f836f-webhook-cert\") pod \"8391846c-50c7-46a0-9111-187e003f836f\" (UID: \"8391846c-50c7-46a0-9111-187e003f836f\") " Jul 22 16:54:29.265661 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.265572 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcct8\" (UniqueName: \"kubernetes.io/projected/8391846c-50c7-46a0-9111-187e003f836f-kube-api-access-xcct8\") pod \"8391846c-50c7-46a0-9111-187e003f836f\" (UID: \"8391846c-50c7-46a0-9111-187e003f836f\") " Jul 22 16:54:29.267552 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.267521 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8391846c-50c7-46a0-9111-187e003f836f-kube-api-access-xcct8" (OuterVolumeSpecName: "kube-api-access-xcct8") pod "8391846c-50c7-46a0-9111-187e003f836f" (UID: "8391846c-50c7-46a0-9111-187e003f836f"). InnerVolumeSpecName "kube-api-access-xcct8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:54:29.267684 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.267618 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8391846c-50c7-46a0-9111-187e003f836f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8391846c-50c7-46a0-9111-187e003f836f" (UID: "8391846c-50c7-46a0-9111-187e003f836f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:54:29.366923 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.366888 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcct8\" (UniqueName: \"kubernetes.io/projected/8391846c-50c7-46a0-9111-187e003f836f-kube-api-access-xcct8\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:54:29.366923 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.366919 2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8391846c-50c7-46a0-9111-187e003f836f-webhook-cert\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:54:29.512300 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.512268 2557 generic.go:356] "Generic (PLEG): container finished" podID="8391846c-50c7-46a0-9111-187e003f836f" containerID="7603e8bb02ad60da8dc9c3ecc0960b9b640d0fc9beb0db6cbbb12e6a984f5f90" exitCode=0 Jul 22 16:54:29.512472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.512316 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58867cc484-f7fqt" Jul 22 16:54:29.512472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.512365 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58867cc484-f7fqt" event={"ID":"8391846c-50c7-46a0-9111-187e003f836f","Type":"ContainerDied","Data":"7603e8bb02ad60da8dc9c3ecc0960b9b640d0fc9beb0db6cbbb12e6a984f5f90"} Jul 22 16:54:29.512472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.512414 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58867cc484-f7fqt" event={"ID":"8391846c-50c7-46a0-9111-187e003f836f","Type":"ContainerDied","Data":"3c8528be16d5474ac67d2bf019c25da7908ef42706b021fa019fb76284708b28"} Jul 22 16:54:29.512472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.512436 2557 scope.go:117] "RemoveContainer" containerID="7603e8bb02ad60da8dc9c3ecc0960b9b640d0fc9beb0db6cbbb12e6a984f5f90" Jul 22 16:54:29.522154 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.522132 2557 scope.go:117] "RemoveContainer" containerID="7603e8bb02ad60da8dc9c3ecc0960b9b640d0fc9beb0db6cbbb12e6a984f5f90" Jul 22 16:54:29.522399 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:54:29.522382 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7603e8bb02ad60da8dc9c3ecc0960b9b640d0fc9beb0db6cbbb12e6a984f5f90\": container with ID starting with 7603e8bb02ad60da8dc9c3ecc0960b9b640d0fc9beb0db6cbbb12e6a984f5f90 not found: ID does not exist" containerID="7603e8bb02ad60da8dc9c3ecc0960b9b640d0fc9beb0db6cbbb12e6a984f5f90" Jul 22 16:54:29.522508 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.522406 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7603e8bb02ad60da8dc9c3ecc0960b9b640d0fc9beb0db6cbbb12e6a984f5f90"} err="failed to get container status \"7603e8bb02ad60da8dc9c3ecc0960b9b640d0fc9beb0db6cbbb12e6a984f5f90\": rpc error: code = NotFound desc = could not find container \"7603e8bb02ad60da8dc9c3ecc0960b9b640d0fc9beb0db6cbbb12e6a984f5f90\": container with ID starting with 7603e8bb02ad60da8dc9c3ecc0960b9b640d0fc9beb0db6cbbb12e6a984f5f90 not found: ID does not exist" Jul 22 16:54:29.535436 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.535413 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58867cc484-f7fqt"] Jul 22 16:54:29.539543 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.539513 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58867cc484-f7fqt"] Jul 22 16:54:29.575567 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:29.575542 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8391846c-50c7-46a0-9111-187e003f836f" path="/var/lib/kubelet/pods/8391846c-50c7-46a0-9111-187e003f836f/volumes" Jul 22 16:54:36.734048 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:54:36.733976 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:55:21.183662 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:21.183622 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:55:21.283919 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:21.283886 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:55:30.446312 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.446231 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58867cc484-vtkb8"] Jul 22 16:55:30.446876 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.446651 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8391846c-50c7-46a0-9111-187e003f836f" containerName="manager" Jul 22 16:55:30.446876 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.446663 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8391846c-50c7-46a0-9111-187e003f836f" containerName="manager" Jul 22 16:55:30.446876 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.446718 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8391846c-50c7-46a0-9111-187e003f836f" containerName="manager" Jul 22 16:55:30.449959 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.449938 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58867cc484-vtkb8" Jul 22 16:55:30.453653 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.453633 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wmd78\"" Jul 22 16:55:30.453653 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.453635 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 16:55:30.453822 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.453655 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 16:55:30.460942 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.460922 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58867cc484-vtkb8"] Jul 22 16:55:30.615068 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.615040 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjqg2\" (UniqueName: \"kubernetes.io/projected/434154b9-a9f0-4993-8c76-1eb665470767-kube-api-access-bjqg2\") pod \"maas-controller-58867cc484-vtkb8\" (UID: \"434154b9-a9f0-4993-8c76-1eb665470767\") " pod="opendatahub/maas-controller-58867cc484-vtkb8" Jul 22 16:55:30.615206 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.615097 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/434154b9-a9f0-4993-8c76-1eb665470767-webhook-cert\") pod \"maas-controller-58867cc484-vtkb8\" (UID: \"434154b9-a9f0-4993-8c76-1eb665470767\") " pod="opendatahub/maas-controller-58867cc484-vtkb8" Jul 22 16:55:30.716524 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.716448 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/434154b9-a9f0-4993-8c76-1eb665470767-webhook-cert\") pod \"maas-controller-58867cc484-vtkb8\" (UID: \"434154b9-a9f0-4993-8c76-1eb665470767\") " pod="opendatahub/maas-controller-58867cc484-vtkb8" Jul 22 16:55:30.716524 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.716515 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjqg2\" (UniqueName: \"kubernetes.io/projected/434154b9-a9f0-4993-8c76-1eb665470767-kube-api-access-bjqg2\") pod \"maas-controller-58867cc484-vtkb8\" (UID: \"434154b9-a9f0-4993-8c76-1eb665470767\") " pod="opendatahub/maas-controller-58867cc484-vtkb8" Jul 22 16:55:30.718998 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.718974 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/434154b9-a9f0-4993-8c76-1eb665470767-webhook-cert\") pod \"maas-controller-58867cc484-vtkb8\" (UID: \"434154b9-a9f0-4993-8c76-1eb665470767\") " pod="opendatahub/maas-controller-58867cc484-vtkb8" Jul 22 16:55:30.725578 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.725554 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjqg2\" (UniqueName: \"kubernetes.io/projected/434154b9-a9f0-4993-8c76-1eb665470767-kube-api-access-bjqg2\") pod \"maas-controller-58867cc484-vtkb8\" (UID: \"434154b9-a9f0-4993-8c76-1eb665470767\") " pod="opendatahub/maas-controller-58867cc484-vtkb8" Jul 22 16:55:30.780005 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.779974 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58867cc484-vtkb8" Jul 22 16:55:30.920429 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:30.920402 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58867cc484-vtkb8"] Jul 22 16:55:30.921976 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:55:30.921944 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434154b9_a9f0_4993_8c76_1eb665470767.slice/crio-6289555a1f3b5a1e8fd06291acc02cdb3cf98c70816906302af5da4e66a947a4 WatchSource:0}: Error finding container 6289555a1f3b5a1e8fd06291acc02cdb3cf98c70816906302af5da4e66a947a4: Status 404 returned error can't find the container with id 6289555a1f3b5a1e8fd06291acc02cdb3cf98c70816906302af5da4e66a947a4 Jul 22 16:55:31.810574 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:31.810536 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58867cc484-vtkb8" event={"ID":"434154b9-a9f0-4993-8c76-1eb665470767","Type":"ContainerStarted","Data":"dfcaabdc9aa1a2b82a9f513d5f98b55d0a63a3c90390c650d3f87f65f453b932"} Jul 22 16:55:31.810951 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:31.810597 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58867cc484-vtkb8" event={"ID":"434154b9-a9f0-4993-8c76-1eb665470767","Type":"ContainerStarted","Data":"6289555a1f3b5a1e8fd06291acc02cdb3cf98c70816906302af5da4e66a947a4"} Jul 22 16:55:31.810951 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:31.810634 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58867cc484-vtkb8" Jul 22 16:55:31.832336 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:31.832291 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58867cc484-vtkb8" podStartSLOduration=1.134358943 podStartE2EDuration="1.832276803s" totalImagesPullingTime="697.91786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:55:31.830481932 +0000 UTC m=+896.807210264" watchObservedRunningTime="2026-07-22 16:55:31.832276803 +0000 UTC m=+896.809005180" Jul 22 16:55:33.610469 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:33.610433 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:55:33.711850 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:33.711814 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:55:42.817457 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:42.817419 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58867cc484-vtkb8" Jul 22 16:55:47.360820 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.360777 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:55:47.465269 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.465230 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58867cc484-vtkb8"] Jul 22 16:55:47.465503 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.465455 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58867cc484-vtkb8" podUID="434154b9-a9f0-4993-8c76-1eb665470767" containerName="manager" containerID="cri-o://dfcaabdc9aa1a2b82a9f513d5f98b55d0a63a3c90390c650d3f87f65f453b932" gracePeriod=30 Jul 22 16:55:47.717296 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.717272 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58867cc484-vtkb8" Jul 22 16:55:47.769848 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.769819 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjqg2\" (UniqueName: \"kubernetes.io/projected/434154b9-a9f0-4993-8c76-1eb665470767-kube-api-access-bjqg2\") pod \"434154b9-a9f0-4993-8c76-1eb665470767\" (UID: \"434154b9-a9f0-4993-8c76-1eb665470767\") " Jul 22 16:55:47.770043 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.769948 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/434154b9-a9f0-4993-8c76-1eb665470767-webhook-cert\") pod \"434154b9-a9f0-4993-8c76-1eb665470767\" (UID: \"434154b9-a9f0-4993-8c76-1eb665470767\") " Jul 22 16:55:47.771941 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.771903 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434154b9-a9f0-4993-8c76-1eb665470767-kube-api-access-bjqg2" (OuterVolumeSpecName: "kube-api-access-bjqg2") pod "434154b9-a9f0-4993-8c76-1eb665470767" (UID: "434154b9-a9f0-4993-8c76-1eb665470767"). InnerVolumeSpecName "kube-api-access-bjqg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 16:55:47.772271 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.772247 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434154b9-a9f0-4993-8c76-1eb665470767-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "434154b9-a9f0-4993-8c76-1eb665470767" (UID: "434154b9-a9f0-4993-8c76-1eb665470767"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 16:55:47.870994 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.870968 2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/434154b9-a9f0-4993-8c76-1eb665470767-webhook-cert\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:55:47.870994 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.870992 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjqg2\" (UniqueName: \"kubernetes.io/projected/434154b9-a9f0-4993-8c76-1eb665470767-kube-api-access-bjqg2\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 16:55:47.888058 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.887960 2557 generic.go:356] "Generic (PLEG): container finished" podID="434154b9-a9f0-4993-8c76-1eb665470767" containerID="dfcaabdc9aa1a2b82a9f513d5f98b55d0a63a3c90390c650d3f87f65f453b932" exitCode=0 Jul 22 16:55:47.888058 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.888010 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58867cc484-vtkb8" Jul 22 16:55:47.888058 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.888010 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58867cc484-vtkb8" event={"ID":"434154b9-a9f0-4993-8c76-1eb665470767","Type":"ContainerDied","Data":"dfcaabdc9aa1a2b82a9f513d5f98b55d0a63a3c90390c650d3f87f65f453b932"} Jul 22 16:55:47.888242 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.888086 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58867cc484-vtkb8" event={"ID":"434154b9-a9f0-4993-8c76-1eb665470767","Type":"ContainerDied","Data":"6289555a1f3b5a1e8fd06291acc02cdb3cf98c70816906302af5da4e66a947a4"} Jul 22 16:55:47.888242 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.888109 2557 scope.go:117] "RemoveContainer" containerID="dfcaabdc9aa1a2b82a9f513d5f98b55d0a63a3c90390c650d3f87f65f453b932" Jul 22 16:55:47.896972 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.896953 2557 scope.go:117] "RemoveContainer" containerID="dfcaabdc9aa1a2b82a9f513d5f98b55d0a63a3c90390c650d3f87f65f453b932" Jul 22 16:55:47.897275 ip-10-0-132-94 kubenswrapper[2557]: E0722 16:55:47.897252 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfcaabdc9aa1a2b82a9f513d5f98b55d0a63a3c90390c650d3f87f65f453b932\": container with ID starting with dfcaabdc9aa1a2b82a9f513d5f98b55d0a63a3c90390c650d3f87f65f453b932 not found: ID does not exist" containerID="dfcaabdc9aa1a2b82a9f513d5f98b55d0a63a3c90390c650d3f87f65f453b932" Jul 22 16:55:47.897369 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.897282 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfcaabdc9aa1a2b82a9f513d5f98b55d0a63a3c90390c650d3f87f65f453b932"} err="failed to get container status \"dfcaabdc9aa1a2b82a9f513d5f98b55d0a63a3c90390c650d3f87f65f453b932\": rpc error: code = NotFound desc = could not find container \"dfcaabdc9aa1a2b82a9f513d5f98b55d0a63a3c90390c650d3f87f65f453b932\": container with ID starting with dfcaabdc9aa1a2b82a9f513d5f98b55d0a63a3c90390c650d3f87f65f453b932 not found: ID does not exist" Jul 22 16:55:47.910946 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.910919 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58867cc484-vtkb8"] Jul 22 16:55:47.914685 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:47.914664 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58867cc484-vtkb8"] Jul 22 16:55:48.646831 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.646796 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:55:48.709111 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.709078 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58867cc484-6m6hv"] Jul 22 16:55:48.709472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.709454 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="434154b9-a9f0-4993-8c76-1eb665470767" containerName="manager" Jul 22 16:55:48.709472 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.709467 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="434154b9-a9f0-4993-8c76-1eb665470767" containerName="manager" Jul 22 16:55:48.709566 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.709530 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="434154b9-a9f0-4993-8c76-1eb665470767" containerName="manager" Jul 22 16:55:48.713711 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.713691 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58867cc484-6m6hv" Jul 22 16:55:48.717297 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.717276 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 16:55:48.717406 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.717307 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 16:55:48.717406 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.717320 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wmd78\"" Jul 22 16:55:48.722405 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.722368 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58867cc484-6m6hv"] Jul 22 16:55:48.778418 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.778382 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e98623d2-9137-4308-bf23-93cba840f19a-webhook-cert\") pod \"maas-controller-58867cc484-6m6hv\" (UID: \"e98623d2-9137-4308-bf23-93cba840f19a\") " pod="opendatahub/maas-controller-58867cc484-6m6hv" Jul 22 16:55:48.778603 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.778479 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkz2n\" (UniqueName: \"kubernetes.io/projected/e98623d2-9137-4308-bf23-93cba840f19a-kube-api-access-dkz2n\") pod \"maas-controller-58867cc484-6m6hv\" (UID: \"e98623d2-9137-4308-bf23-93cba840f19a\") " pod="opendatahub/maas-controller-58867cc484-6m6hv" Jul 22 16:55:48.878968 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.878936 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e98623d2-9137-4308-bf23-93cba840f19a-webhook-cert\") pod \"maas-controller-58867cc484-6m6hv\" (UID: \"e98623d2-9137-4308-bf23-93cba840f19a\") " pod="opendatahub/maas-controller-58867cc484-6m6hv" Jul 22 16:55:48.879148 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.878985 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkz2n\" (UniqueName: \"kubernetes.io/projected/e98623d2-9137-4308-bf23-93cba840f19a-kube-api-access-dkz2n\") pod \"maas-controller-58867cc484-6m6hv\" (UID: \"e98623d2-9137-4308-bf23-93cba840f19a\") " pod="opendatahub/maas-controller-58867cc484-6m6hv" Jul 22 16:55:48.881477 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.881444 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e98623d2-9137-4308-bf23-93cba840f19a-webhook-cert\") pod \"maas-controller-58867cc484-6m6hv\" (UID: \"e98623d2-9137-4308-bf23-93cba840f19a\") " pod="opendatahub/maas-controller-58867cc484-6m6hv" Jul 22 16:55:48.887870 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:48.887841 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkz2n\" (UniqueName: \"kubernetes.io/projected/e98623d2-9137-4308-bf23-93cba840f19a-kube-api-access-dkz2n\") pod \"maas-controller-58867cc484-6m6hv\" (UID: \"e98623d2-9137-4308-bf23-93cba840f19a\") " pod="opendatahub/maas-controller-58867cc484-6m6hv" Jul 22 16:55:49.025183 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:49.025153 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58867cc484-6m6hv" Jul 22 16:55:49.159388 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:49.159362 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58867cc484-6m6hv"] Jul 22 16:55:49.161282 ip-10-0-132-94 kubenswrapper[2557]: W0722 16:55:49.161255 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98623d2_9137_4308_bf23_93cba840f19a.slice/crio-3a11d6720a09c68ce5ea965cd0f79d26f66b529a131742c2e206e2b68b4cec65 WatchSource:0}: Error finding container 3a11d6720a09c68ce5ea965cd0f79d26f66b529a131742c2e206e2b68b4cec65: Status 404 returned error can't find the container with id 3a11d6720a09c68ce5ea965cd0f79d26f66b529a131742c2e206e2b68b4cec65 Jul 22 16:55:49.574708 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:49.574686 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="434154b9-a9f0-4993-8c76-1eb665470767" path="/var/lib/kubelet/pods/434154b9-a9f0-4993-8c76-1eb665470767/volumes" Jul 22 16:55:49.905854 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:49.905763 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58867cc484-6m6hv" event={"ID":"e98623d2-9137-4308-bf23-93cba840f19a","Type":"ContainerStarted","Data":"637f768db68f86b72c4efbce1dd36add6da26071e38b34b68240bfe90848468c"} Jul 22 16:55:49.905854 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:49.905849 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58867cc484-6m6hv" event={"ID":"e98623d2-9137-4308-bf23-93cba840f19a","Type":"ContainerStarted","Data":"3a11d6720a09c68ce5ea965cd0f79d26f66b529a131742c2e206e2b68b4cec65"} Jul 22 16:55:49.906341 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:49.905919 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58867cc484-6m6hv" Jul 22 16:55:49.931166 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:55:49.931109 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58867cc484-6m6hv" podStartSLOduration=1.524418179 podStartE2EDuration="1.931094284s" totalImagesPullingTime="406.676105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 16:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 16:55:49.92995915 +0000 UTC m=+914.906687463" watchObservedRunningTime="2026-07-22 16:55:49.931094284 +0000 UTC m=+914.907822599" Jul 22 16:56:00.913254 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:56:00.913221 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58867cc484-6m6hv" Jul 22 16:56:29.328871 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:56:29.328832 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:56:29.420973 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:56:29.420939 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:56:29.521408 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:56:29.521365 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:56:35.473317 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:56:35.473279 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:56:35.532122 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:56:35.532088 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:56:35.609291 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:56:35.609257 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:56:35.709767 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:56:35.709726 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:56:35.832747 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:56:35.832713 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:56:41.666387 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:56:41.666352 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:56:41.745057 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:56:41.745002 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 16:58:54.695113 ip-10-0-132-94 kubenswrapper[2557]: I0722 16:58:54.695029 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:00:00.147874 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:00:00.147792 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745660-7w47l"] Jul 22 17:00:00.151200 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:00:00.151183 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745660-7w47l" Jul 22 17:00:00.153764 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:00:00.153742 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vln7b\"" Jul 22 17:00:00.161325 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:00:00.161302 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745660-7w47l" Jul 22 17:00:00.168103 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:00:00.168080 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745660-7w47l"] Jul 22 17:00:00.297158 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:00:00.297136 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745660-7w47l"] Jul 22 17:00:00.299314 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:00:00.299278 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61361050_4222_4246_8b76_678fc9638e58.slice/crio-541e3e74adc4572fe1e9db403b1e73b437e0473504f70f6d5b303406b10f9f8b WatchSource:0}: Error finding container 541e3e74adc4572fe1e9db403b1e73b437e0473504f70f6d5b303406b10f9f8b: Status 404 returned error can't find the container with id 541e3e74adc4572fe1e9db403b1e73b437e0473504f70f6d5b303406b10f9f8b Jul 22 17:00:00.301216 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:00:00.301199 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 17:00:01.105245 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:00:01.105206 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745660-7w47l" event={"ID":"61361050-4222-4246-8b76-678fc9638e58","Type":"ContainerStarted","Data":"541e3e74adc4572fe1e9db403b1e73b437e0473504f70f6d5b303406b10f9f8b"} Jul 22 17:00:04.128524 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:00:04.128431 2557 generic.go:356] "Generic (PLEG): container finished" podID="61361050-4222-4246-8b76-678fc9638e58" containerID="f9d801e2aeaa9fb6fdfd1b6ec53cc5fbc80b6b08e0062306a8f1efb927b5070b" exitCode=0 Jul 22 17:00:04.128524 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:00:04.128476 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745660-7w47l" event={"ID":"61361050-4222-4246-8b76-678fc9638e58","Type":"ContainerDied","Data":"f9d801e2aeaa9fb6fdfd1b6ec53cc5fbc80b6b08e0062306a8f1efb927b5070b"} Jul 22 17:00:05.265420 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:00:05.265392 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745660-7w47l" Jul 22 17:00:06.138625 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:00:06.138596 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745660-7w47l" Jul 22 17:00:06.138800 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:00:06.138589 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745660-7w47l" event={"ID":"61361050-4222-4246-8b76-678fc9638e58","Type":"ContainerDied","Data":"541e3e74adc4572fe1e9db403b1e73b437e0473504f70f6d5b303406b10f9f8b"} Jul 22 17:00:06.138800 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:00:06.138720 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="541e3e74adc4572fe1e9db403b1e73b437e0473504f70f6d5b303406b10f9f8b" Jul 22 17:01:04.701018 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:01:04.700983 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:01:06.219365 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:01:06.219328 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:04:16.460634 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:04:16.460591 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:04:21.081435 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:04:21.081398 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:04:22.806957 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:04:22.806919 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:04:49.519475 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:04:49.519359 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:04:49.908556 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:04:49.908478 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:04:55.480338 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:04:55.480300 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:05:28.180159 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:05:28.180123 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:04.231491 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:04.231405 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:29.828140 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:29.828094 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:32.939063 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:32.939014 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:33.038710 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:33.038657 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:33.473361 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:33.473328 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:39.470659 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:39.470623 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:39.571977 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:39.571946 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:39.689492 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:39.689461 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:40.065278 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:40.065246 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:40.399741 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:40.399655 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:42.715508 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:42.715478 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:42.801112 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:42.801069 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:42.807117 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:42.807088 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg"] Jul 22 17:08:42.807512 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:42.807497 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61361050-4222-4246-8b76-678fc9638e58" containerName="cleanup" Jul 22 17:08:42.807561 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:42.807513 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="61361050-4222-4246-8b76-678fc9638e58" containerName="cleanup" Jul 22 17:08:42.807603 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:42.807583 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="61361050-4222-4246-8b76-678fc9638e58" containerName="cleanup" Jul 22 17:08:42.810872 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:42.810856 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:08:42.813499 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:42.813474 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8sp7j\"" Jul 22 17:08:42.813767 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:42.813749 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-76a9cf43-serving-cert\"" Jul 22 17:08:42.823094 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:42.823066 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg"] Jul 22 17:08:42.868264 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:42.868232 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:42.955923 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:42.955884 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5303b1-045b-49e1-bdd9-560d1549d9d0-maas-api-tls\") pod \"maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg\" (UID: \"db5303b1-045b-49e1-bdd9-560d1549d9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:08:42.956148 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:42.955935 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5303b1-045b-49e1-bdd9-560d1549d9d0-openshift-service-ca\") pod \"maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg\" (UID: \"db5303b1-045b-49e1-bdd9-560d1549d9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:08:42.956148 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:42.956054 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6jgm\" (UniqueName: \"kubernetes.io/projected/db5303b1-045b-49e1-bdd9-560d1549d9d0-kube-api-access-l6jgm\") pod \"maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg\" (UID: \"db5303b1-045b-49e1-bdd9-560d1549d9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:08:43.057204 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:43.057163 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5303b1-045b-49e1-bdd9-560d1549d9d0-maas-api-tls\") pod \"maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg\" (UID: \"db5303b1-045b-49e1-bdd9-560d1549d9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:08:43.057421 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:43.057216 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5303b1-045b-49e1-bdd9-560d1549d9d0-openshift-service-ca\") pod \"maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg\" (UID: \"db5303b1-045b-49e1-bdd9-560d1549d9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:08:43.057421 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:43.057274 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6jgm\" (UniqueName: \"kubernetes.io/projected/db5303b1-045b-49e1-bdd9-560d1549d9d0-kube-api-access-l6jgm\") pod \"maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg\" (UID: \"db5303b1-045b-49e1-bdd9-560d1549d9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:08:43.057421 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:08:43.057309 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-serving-cert: secret "maas-api-e2e-ait-76a9cf43-serving-cert" not found Jul 22 17:08:43.057421 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:08:43.057383 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db5303b1-045b-49e1-bdd9-560d1549d9d0-maas-api-tls podName:db5303b1-045b-49e1-bdd9-560d1549d9d0 nodeName:}" failed. No retries permitted until 2026-07-22 17:08:43.557365878 +0000 UTC m=+1688.534094175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db5303b1-045b-49e1-bdd9-560d1549d9d0-maas-api-tls") pod "maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" (UID: "db5303b1-045b-49e1-bdd9-560d1549d9d0") : secret "maas-api-e2e-ait-76a9cf43-serving-cert" not found Jul 22 17:08:43.057879 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:43.057859 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5303b1-045b-49e1-bdd9-560d1549d9d0-openshift-service-ca\") pod \"maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg\" (UID: \"db5303b1-045b-49e1-bdd9-560d1549d9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:08:43.070561 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:43.070536 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6jgm\" (UniqueName: \"kubernetes.io/projected/db5303b1-045b-49e1-bdd9-560d1549d9d0-kube-api-access-l6jgm\") pod \"maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg\" (UID: \"db5303b1-045b-49e1-bdd9-560d1549d9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:08:43.320721 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:43.320639 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:43.424575 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:43.424544 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:43.529195 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:43.529165 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:08:43.562904 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:43.562871 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5303b1-045b-49e1-bdd9-560d1549d9d0-maas-api-tls\") pod \"maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg\" (UID: \"db5303b1-045b-49e1-bdd9-560d1549d9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:08:43.565306 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:43.565285 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5303b1-045b-49e1-bdd9-560d1549d9d0-maas-api-tls\") pod \"maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg\" (UID: \"db5303b1-045b-49e1-bdd9-560d1549d9d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:08:43.722098 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:43.722000 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:08:43.866828 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:43.866795 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg"] Jul 22 17:08:43.870174 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:08:43.870147 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5303b1_045b_49e1_bdd9_560d1549d9d0.slice/crio-4118b2f37d5f7dfafb3408b0c81dc3c80abcfa59cad25b6a409fe98b046efce2 WatchSource:0}: Error finding container 4118b2f37d5f7dfafb3408b0c81dc3c80abcfa59cad25b6a409fe98b046efce2: Status 404 returned error can't find the container with id 4118b2f37d5f7dfafb3408b0c81dc3c80abcfa59cad25b6a409fe98b046efce2 Jul 22 17:08:43.871796 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:43.871780 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 17:08:44.622306 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:44.622213 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" event={"ID":"db5303b1-045b-49e1-bdd9-560d1549d9d0","Type":"ContainerStarted","Data":"60cc7ae13ed73b07269fd0e268c14365acc767da2d3dc04e1c9ec630c41c4db8"} Jul 22 17:08:44.622306 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:44.622261 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" event={"ID":"db5303b1-045b-49e1-bdd9-560d1549d9d0","Type":"ContainerStarted","Data":"4118b2f37d5f7dfafb3408b0c81dc3c80abcfa59cad25b6a409fe98b046efce2"} Jul 22 17:08:44.622526 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:44.622343 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:08:50.633884 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:50.633853 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:08:50.654610 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:08:50.654556 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" podStartSLOduration=8.190421043 podStartE2EDuration="8.654540276s" totalImagesPullingTime="464.119233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:08:44.663379886 +0000 UTC m=+1689.640108201" watchObservedRunningTime="2026-07-22 17:08:50.654540276 +0000 UTC m=+1695.631268592" Jul 22 17:09:08.225892 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:08.225854 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl"] Jul 22 17:09:08.229505 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:08.229488 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl" Jul 22 17:09:08.232321 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:08.232300 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vln7b\"" Jul 22 17:09:08.240830 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:08.240807 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl"] Jul 22 17:09:08.295926 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:08.295887 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a24df3a-a7c0-466a-9c18-4d25e67d3b9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl\" (UID: \"6a24df3a-a7c0-466a-9c18-4d25e67d3b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl" Jul 22 17:09:08.397380 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:08.397343 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a24df3a-a7c0-466a-9c18-4d25e67d3b9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl\" (UID: \"6a24df3a-a7c0-466a-9c18-4d25e67d3b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl" Jul 22 17:09:08.397931 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:08.397911 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a24df3a-a7c0-466a-9c18-4d25e67d3b9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl\" (UID: \"6a24df3a-a7c0-466a-9c18-4d25e67d3b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl" Jul 22 17:09:08.541260 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:08.541166 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl" Jul 22 17:09:08.677841 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:08.677805 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl"] Jul 22 17:09:08.679529 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:09:08.679494 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a24df3a_a7c0_466a_9c18_4d25e67d3b9e.slice/crio-884f5174ba739b8da6e9dd46c681bcdac276d1ce82f774d68afd010c03e6ae1a WatchSource:0}: Error finding container 884f5174ba739b8da6e9dd46c681bcdac276d1ce82f774d68afd010c03e6ae1a: Status 404 returned error can't find the container with id 884f5174ba739b8da6e9dd46c681bcdac276d1ce82f774d68afd010c03e6ae1a Jul 22 17:09:08.745330 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:08.745294 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl" event={"ID":"6a24df3a-a7c0-466a-9c18-4d25e67d3b9e","Type":"ContainerStarted","Data":"884f5174ba739b8da6e9dd46c681bcdac276d1ce82f774d68afd010c03e6ae1a"} Jul 22 17:09:09.752988 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:09.752952 2557 generic.go:356] "Generic (PLEG): container finished" podID="6a24df3a-a7c0-466a-9c18-4d25e67d3b9e" containerID="7025f63a9e4169b7716a826e3ee571394d43605aa30d1d8dcbbdcfbdbd833999" exitCode=0 Jul 22 17:09:09.753400 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:09.753050 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl" event={"ID":"6a24df3a-a7c0-466a-9c18-4d25e67d3b9e","Type":"ContainerDied","Data":"7025f63a9e4169b7716a826e3ee571394d43605aa30d1d8dcbbdcfbdbd833999"} Jul 22 17:09:10.885705 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:10.885681 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl" Jul 22 17:09:11.019754 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:11.019675 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a24df3a-a7c0-466a-9c18-4d25e67d3b9e-maas-api-service-ca\") pod \"6a24df3a-a7c0-466a-9c18-4d25e67d3b9e\" (UID: \"6a24df3a-a7c0-466a-9c18-4d25e67d3b9e\") " Jul 22 17:09:11.020097 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:11.020069 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a24df3a-a7c0-466a-9c18-4d25e67d3b9e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a24df3a-a7c0-466a-9c18-4d25e67d3b9e" (UID: "6a24df3a-a7c0-466a-9c18-4d25e67d3b9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:09:11.121184 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:11.121151 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a24df3a-a7c0-466a-9c18-4d25e67d3b9e-maas-api-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:09:11.764642 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:11.764614 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl" Jul 22 17:09:11.764808 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:11.764611 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl" event={"ID":"6a24df3a-a7c0-466a-9c18-4d25e67d3b9e","Type":"ContainerDied","Data":"884f5174ba739b8da6e9dd46c681bcdac276d1ce82f774d68afd010c03e6ae1a"} Jul 22 17:09:11.764808 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:11.764739 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="884f5174ba739b8da6e9dd46c681bcdac276d1ce82f774d68afd010c03e6ae1a" Jul 22 17:09:18.754511 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:18.754475 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg"] Jul 22 17:09:18.755096 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:18.754700 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" podUID="db5303b1-045b-49e1-bdd9-560d1549d9d0" containerName="maas-api" containerID="cri-o://60cc7ae13ed73b07269fd0e268c14365acc767da2d3dc04e1c9ec630c41c4db8" gracePeriod=30 Jul 22 17:09:18.782278 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:18.782249 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:09:19.007287 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.007215 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:09:19.090856 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.090823 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6jgm\" (UniqueName: \"kubernetes.io/projected/db5303b1-045b-49e1-bdd9-560d1549d9d0-kube-api-access-l6jgm\") pod \"db5303b1-045b-49e1-bdd9-560d1549d9d0\" (UID: \"db5303b1-045b-49e1-bdd9-560d1549d9d0\") " Jul 22 17:09:19.091064 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.090890 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5303b1-045b-49e1-bdd9-560d1549d9d0-openshift-service-ca\") pod \"db5303b1-045b-49e1-bdd9-560d1549d9d0\" (UID: \"db5303b1-045b-49e1-bdd9-560d1549d9d0\") " Jul 22 17:09:19.091064 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.090922 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5303b1-045b-49e1-bdd9-560d1549d9d0-maas-api-tls\") pod \"db5303b1-045b-49e1-bdd9-560d1549d9d0\" (UID: \"db5303b1-045b-49e1-bdd9-560d1549d9d0\") " Jul 22 17:09:19.091305 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.091269 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5303b1-045b-49e1-bdd9-560d1549d9d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "db5303b1-045b-49e1-bdd9-560d1549d9d0" (UID: "db5303b1-045b-49e1-bdd9-560d1549d9d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:09:19.093168 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.093135 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5303b1-045b-49e1-bdd9-560d1549d9d0-kube-api-access-l6jgm" (OuterVolumeSpecName: "kube-api-access-l6jgm") pod "db5303b1-045b-49e1-bdd9-560d1549d9d0" (UID: "db5303b1-045b-49e1-bdd9-560d1549d9d0"). InnerVolumeSpecName "kube-api-access-l6jgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:09:19.093168 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.093141 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db5303b1-045b-49e1-bdd9-560d1549d9d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "db5303b1-045b-49e1-bdd9-560d1549d9d0" (UID: "db5303b1-045b-49e1-bdd9-560d1549d9d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:09:19.191839 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.191807 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6jgm\" (UniqueName: \"kubernetes.io/projected/db5303b1-045b-49e1-bdd9-560d1549d9d0-kube-api-access-l6jgm\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:09:19.191839 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.191835 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5303b1-045b-49e1-bdd9-560d1549d9d0-openshift-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:09:19.191839 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.191845 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db5303b1-045b-49e1-bdd9-560d1549d9d0-maas-api-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:09:19.804641 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.804604 2557 generic.go:356] "Generic (PLEG): container finished" podID="db5303b1-045b-49e1-bdd9-560d1549d9d0" containerID="60cc7ae13ed73b07269fd0e268c14365acc767da2d3dc04e1c9ec630c41c4db8" exitCode=0 Jul 22 17:09:19.804641 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.804642 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" event={"ID":"db5303b1-045b-49e1-bdd9-560d1549d9d0","Type":"ContainerDied","Data":"60cc7ae13ed73b07269fd0e268c14365acc767da2d3dc04e1c9ec630c41c4db8"} Jul 22 17:09:19.805164 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.804651 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" Jul 22 17:09:19.805164 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.804684 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg" event={"ID":"db5303b1-045b-49e1-bdd9-560d1549d9d0","Type":"ContainerDied","Data":"4118b2f37d5f7dfafb3408b0c81dc3c80abcfa59cad25b6a409fe98b046efce2"} Jul 22 17:09:19.805164 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.804706 2557 scope.go:117] "RemoveContainer" containerID="60cc7ae13ed73b07269fd0e268c14365acc767da2d3dc04e1c9ec630c41c4db8" Jul 22 17:09:19.813352 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.813336 2557 scope.go:117] "RemoveContainer" containerID="60cc7ae13ed73b07269fd0e268c14365acc767da2d3dc04e1c9ec630c41c4db8" Jul 22 17:09:19.813574 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:09:19.813556 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60cc7ae13ed73b07269fd0e268c14365acc767da2d3dc04e1c9ec630c41c4db8\": container with ID starting with 60cc7ae13ed73b07269fd0e268c14365acc767da2d3dc04e1c9ec630c41c4db8 not found: ID does not exist" containerID="60cc7ae13ed73b07269fd0e268c14365acc767da2d3dc04e1c9ec630c41c4db8" Jul 22 17:09:19.813633 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.813588 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60cc7ae13ed73b07269fd0e268c14365acc767da2d3dc04e1c9ec630c41c4db8"} err="failed to get container status \"60cc7ae13ed73b07269fd0e268c14365acc767da2d3dc04e1c9ec630c41c4db8\": rpc error: code = NotFound desc = could not find container \"60cc7ae13ed73b07269fd0e268c14365acc767da2d3dc04e1c9ec630c41c4db8\": container with ID starting with 60cc7ae13ed73b07269fd0e268c14365acc767da2d3dc04e1c9ec630c41c4db8 not found: ID does not exist" Jul 22 17:09:19.823177 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.823151 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg"] Jul 22 17:09:19.827328 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:19.827307 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-76a9cf43-798b567dfc-pqktg"] Jul 22 17:09:21.575548 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:21.575512 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db5303b1-045b-49e1-bdd9-560d1549d9d0" path="/var/lib/kubelet/pods/db5303b1-045b-49e1-bdd9-560d1549d9d0/volumes" Jul 22 17:09:23.312705 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:23.312666 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:09:23.313498 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:23.313462 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl"] Jul 22 17:09:23.315685 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:23.315661 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a9cf43-2rvbl"] Jul 22 17:09:23.503549 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:23.503518 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:09:23.575489 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:23.575403 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a24df3a-a7c0-466a-9c18-4d25e67d3b9e" path="/var/lib/kubelet/pods/6a24df3a-a7c0-466a-9c18-4d25e67d3b9e/volumes" Jul 22 17:09:23.757833 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:23.757804 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:09:24.063005 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:24.062970 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:09:24.156011 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:24.155977 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:09:24.329356 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:24.329267 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:09:24.808090 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:24.808055 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:09:25.078167 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:25.078087 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:09:27.223243 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.223210 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:09:27.463392 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.463358 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr"] Jul 22 17:09:27.463827 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.463810 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db5303b1-045b-49e1-bdd9-560d1549d9d0" containerName="maas-api" Jul 22 17:09:27.463827 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.463826 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="db5303b1-045b-49e1-bdd9-560d1549d9d0" containerName="maas-api" Jul 22 17:09:27.463994 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.463834 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a24df3a-a7c0-466a-9c18-4d25e67d3b9e" containerName="revoke-keys" Jul 22 17:09:27.463994 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.463840 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a24df3a-a7c0-466a-9c18-4d25e67d3b9e" containerName="revoke-keys" Jul 22 17:09:27.463994 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.463900 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a24df3a-a7c0-466a-9c18-4d25e67d3b9e" containerName="revoke-keys" Jul 22 17:09:27.463994 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.463910 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="db5303b1-045b-49e1-bdd9-560d1549d9d0" containerName="maas-api" Jul 22 17:09:27.468538 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.468509 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" Jul 22 17:09:27.471294 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.471265 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-af56de08-serving-cert\"" Jul 22 17:09:27.471816 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.471800 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8sp7j\"" Jul 22 17:09:27.488937 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.488914 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr"] Jul 22 17:09:27.522650 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.522616 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:09:27.569139 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.569103 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa278095-559c-413b-85c4-dff17ec6267e-openshift-service-ca\") pod \"maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr\" (UID: \"fa278095-559c-413b-85c4-dff17ec6267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" Jul 22 17:09:27.569298 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.569152 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnjdz\" (UniqueName: \"kubernetes.io/projected/fa278095-559c-413b-85c4-dff17ec6267e-kube-api-access-lnjdz\") pod \"maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr\" (UID: \"fa278095-559c-413b-85c4-dff17ec6267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" Jul 22 17:09:27.569298 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.569202 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa278095-559c-413b-85c4-dff17ec6267e-maas-api-tls\") pod \"maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr\" (UID: \"fa278095-559c-413b-85c4-dff17ec6267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" Jul 22 17:09:27.669996 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.669965 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa278095-559c-413b-85c4-dff17ec6267e-openshift-service-ca\") pod \"maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr\" (UID: \"fa278095-559c-413b-85c4-dff17ec6267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" Jul 22 17:09:27.670198 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.670011 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnjdz\" (UniqueName: \"kubernetes.io/projected/fa278095-559c-413b-85c4-dff17ec6267e-kube-api-access-lnjdz\") pod \"maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr\" (UID: \"fa278095-559c-413b-85c4-dff17ec6267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" Jul 22 17:09:27.670198 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.670072 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa278095-559c-413b-85c4-dff17ec6267e-maas-api-tls\") pod \"maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr\" (UID: \"fa278095-559c-413b-85c4-dff17ec6267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" Jul 22 17:09:27.670617 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.670587 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa278095-559c-413b-85c4-dff17ec6267e-openshift-service-ca\") pod \"maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr\" (UID: \"fa278095-559c-413b-85c4-dff17ec6267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" Jul 22 17:09:27.672441 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.672420 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa278095-559c-413b-85c4-dff17ec6267e-maas-api-tls\") pod \"maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr\" (UID: \"fa278095-559c-413b-85c4-dff17ec6267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" Jul 22 17:09:27.688341 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.688317 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnjdz\" (UniqueName: \"kubernetes.io/projected/fa278095-559c-413b-85c4-dff17ec6267e-kube-api-access-lnjdz\") pod \"maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr\" (UID: \"fa278095-559c-413b-85c4-dff17ec6267e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" Jul 22 17:09:27.694378 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.694340 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:09:27.779156 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.779128 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" Jul 22 17:09:27.938615 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:27.937119 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr"] Jul 22 17:09:27.941808 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:09:27.941358 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa278095_559c_413b_85c4_dff17ec6267e.slice/crio-e35eb51cde1f9552cbca0cd660718a692424920bb8afbeaaaaa97a6048c64113 WatchSource:0}: Error finding container e35eb51cde1f9552cbca0cd660718a692424920bb8afbeaaaaa97a6048c64113: Status 404 returned error can't find the container with id e35eb51cde1f9552cbca0cd660718a692424920bb8afbeaaaaa97a6048c64113 Jul 22 17:09:28.278780 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:28.278552 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:09:28.382902 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:28.382872 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:09:28.853687 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:28.853651 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" event={"ID":"fa278095-559c-413b-85c4-dff17ec6267e","Type":"ContainerStarted","Data":"4190d727f017290ade42507dda2560f8a07c5840fe01a98a4a44f595bfccfbb1"} Jul 22 17:09:28.853911 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:28.853702 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" event={"ID":"fa278095-559c-413b-85c4-dff17ec6267e","Type":"ContainerStarted","Data":"e35eb51cde1f9552cbca0cd660718a692424920bb8afbeaaaaa97a6048c64113"} Jul 22 17:09:28.853911 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:28.853774 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" Jul 22 17:09:28.888985 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:28.888657 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" podStartSLOduration=1.594239839 podStartE2EDuration="1.888637517s" totalImagesPullingTime="294.397678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:09:28.886909918 +0000 UTC m=+1733.863638235" watchObservedRunningTime="2026-07-22 17:09:28.888637517 +0000 UTC m=+1733.865365824" Jul 22 17:09:34.864269 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:34.864219 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" Jul 22 17:09:52.098087 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:52.098050 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw"] Jul 22 17:09:52.101772 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:52.101754 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw" Jul 22 17:09:52.105001 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:52.104984 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vln7b\"" Jul 22 17:09:52.113847 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:52.113820 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw"] Jul 22 17:09:52.203176 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:52.203138 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff9250-365c-4151-bc45-a4deb625d020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw\" (UID: \"f1ff9250-365c-4151-bc45-a4deb625d020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw" Jul 22 17:09:52.304220 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:52.304186 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff9250-365c-4151-bc45-a4deb625d020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw\" (UID: \"f1ff9250-365c-4151-bc45-a4deb625d020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw" Jul 22 17:09:52.304801 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:52.304780 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff9250-365c-4151-bc45-a4deb625d020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw\" (UID: \"f1ff9250-365c-4151-bc45-a4deb625d020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw" Jul 22 17:09:52.413731 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:52.413639 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw" Jul 22 17:09:52.547486 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:52.547455 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw"] Jul 22 17:09:52.549474 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:09:52.549442 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ff9250_365c_4151_bc45_a4deb625d020.slice/crio-31467f7eecac91833f845eead551cd825cf9e802096d9eb5d70dbd551555a7c6 WatchSource:0}: Error finding container 31467f7eecac91833f845eead551cd825cf9e802096d9eb5d70dbd551555a7c6: Status 404 returned error can't find the container with id 31467f7eecac91833f845eead551cd825cf9e802096d9eb5d70dbd551555a7c6 Jul 22 17:09:52.976303 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:52.976273 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw" event={"ID":"f1ff9250-365c-4151-bc45-a4deb625d020","Type":"ContainerStarted","Data":"5a88607ad4555fbea3f187a2e6f3ba290ef7bf58e03a2ac3fac0de2980396fcf"} Jul 22 17:09:52.976529 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:52.976323 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw" event={"ID":"f1ff9250-365c-4151-bc45-a4deb625d020","Type":"ContainerStarted","Data":"31467f7eecac91833f845eead551cd825cf9e802096d9eb5d70dbd551555a7c6"} Jul 22 17:09:52.996729 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:52.996673 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw" podStartSLOduration=0.996657198 podStartE2EDuration="996.657198ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:09:52.995799339 +0000 UTC m=+1757.972527659" watchObservedRunningTime="2026-07-22 17:09:52.996657198 +0000 UTC m=+1757.973385514" Jul 22 17:09:53.983421 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:53.983388 2557 generic.go:356] "Generic (PLEG): container finished" podID="f1ff9250-365c-4151-bc45-a4deb625d020" containerID="5a88607ad4555fbea3f187a2e6f3ba290ef7bf58e03a2ac3fac0de2980396fcf" exitCode=0 Jul 22 17:09:53.983794 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:53.983447 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw" event={"ID":"f1ff9250-365c-4151-bc45-a4deb625d020","Type":"ContainerDied","Data":"5a88607ad4555fbea3f187a2e6f3ba290ef7bf58e03a2ac3fac0de2980396fcf"} Jul 22 17:09:55.118364 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:55.118339 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw" Jul 22 17:09:55.229737 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:55.229702 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff9250-365c-4151-bc45-a4deb625d020-maas-api-service-ca\") pod \"f1ff9250-365c-4151-bc45-a4deb625d020\" (UID: \"f1ff9250-365c-4151-bc45-a4deb625d020\") " Jul 22 17:09:55.230083 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:55.230062 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ff9250-365c-4151-bc45-a4deb625d020-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f1ff9250-365c-4151-bc45-a4deb625d020" (UID: "f1ff9250-365c-4151-bc45-a4deb625d020"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:09:55.330767 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:55.330684 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ff9250-365c-4151-bc45-a4deb625d020-maas-api-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:09:55.994150 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:55.994119 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw" event={"ID":"f1ff9250-365c-4151-bc45-a4deb625d020","Type":"ContainerDied","Data":"31467f7eecac91833f845eead551cd825cf9e802096d9eb5d70dbd551555a7c6"} Jul 22 17:09:55.994150 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:55.994145 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw" Jul 22 17:09:55.994357 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:09:55.994175 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31467f7eecac91833f845eead551cd825cf9e802096d9eb5d70dbd551555a7c6" Jul 22 17:10:02.651387 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:02.651331 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr"] Jul 22 17:10:02.651886 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:02.651763 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" podUID="fa278095-559c-413b-85c4-dff17ec6267e" containerName="maas-api" containerID="cri-o://4190d727f017290ade42507dda2560f8a07c5840fe01a98a4a44f595bfccfbb1" gracePeriod=30 Jul 22 17:10:02.668134 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:02.667976 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:02.909957 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:02.909895 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" Jul 22 17:10:02.998942 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:02.998904 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnjdz\" (UniqueName: \"kubernetes.io/projected/fa278095-559c-413b-85c4-dff17ec6267e-kube-api-access-lnjdz\") pod \"fa278095-559c-413b-85c4-dff17ec6267e\" (UID: \"fa278095-559c-413b-85c4-dff17ec6267e\") " Jul 22 17:10:02.999138 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:02.998971 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa278095-559c-413b-85c4-dff17ec6267e-maas-api-tls\") pod \"fa278095-559c-413b-85c4-dff17ec6267e\" (UID: \"fa278095-559c-413b-85c4-dff17ec6267e\") " Jul 22 17:10:02.999138 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:02.999071 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa278095-559c-413b-85c4-dff17ec6267e-openshift-service-ca\") pod \"fa278095-559c-413b-85c4-dff17ec6267e\" (UID: \"fa278095-559c-413b-85c4-dff17ec6267e\") " Jul 22 17:10:02.999457 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:02.999426 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa278095-559c-413b-85c4-dff17ec6267e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fa278095-559c-413b-85c4-dff17ec6267e" (UID: "fa278095-559c-413b-85c4-dff17ec6267e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:10:03.001083 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:03.001063 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa278095-559c-413b-85c4-dff17ec6267e-kube-api-access-lnjdz" (OuterVolumeSpecName: "kube-api-access-lnjdz") pod "fa278095-559c-413b-85c4-dff17ec6267e" (UID: "fa278095-559c-413b-85c4-dff17ec6267e"). InnerVolumeSpecName "kube-api-access-lnjdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:10:03.001181 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:03.001162 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa278095-559c-413b-85c4-dff17ec6267e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fa278095-559c-413b-85c4-dff17ec6267e" (UID: "fa278095-559c-413b-85c4-dff17ec6267e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:10:03.032827 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:03.032790 2557 generic.go:356] "Generic (PLEG): container finished" podID="fa278095-559c-413b-85c4-dff17ec6267e" containerID="4190d727f017290ade42507dda2560f8a07c5840fe01a98a4a44f595bfccfbb1" exitCode=0 Jul 22 17:10:03.032979 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:03.032836 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" Jul 22 17:10:03.032979 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:03.032874 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" event={"ID":"fa278095-559c-413b-85c4-dff17ec6267e","Type":"ContainerDied","Data":"4190d727f017290ade42507dda2560f8a07c5840fe01a98a4a44f595bfccfbb1"} Jul 22 17:10:03.032979 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:03.032933 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr" event={"ID":"fa278095-559c-413b-85c4-dff17ec6267e","Type":"ContainerDied","Data":"e35eb51cde1f9552cbca0cd660718a692424920bb8afbeaaaaa97a6048c64113"} Jul 22 17:10:03.032979 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:03.032955 2557 scope.go:117] "RemoveContainer" containerID="4190d727f017290ade42507dda2560f8a07c5840fe01a98a4a44f595bfccfbb1" Jul 22 17:10:03.042602 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:03.042583 2557 scope.go:117] "RemoveContainer" containerID="4190d727f017290ade42507dda2560f8a07c5840fe01a98a4a44f595bfccfbb1" Jul 22 17:10:03.042840 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:10:03.042822 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4190d727f017290ade42507dda2560f8a07c5840fe01a98a4a44f595bfccfbb1\": container with ID starting with 4190d727f017290ade42507dda2560f8a07c5840fe01a98a4a44f595bfccfbb1 not found: ID does not exist" containerID="4190d727f017290ade42507dda2560f8a07c5840fe01a98a4a44f595bfccfbb1" Jul 22 17:10:03.042883 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:03.042847 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4190d727f017290ade42507dda2560f8a07c5840fe01a98a4a44f595bfccfbb1"} err="failed to get container status \"4190d727f017290ade42507dda2560f8a07c5840fe01a98a4a44f595bfccfbb1\": rpc error: code = NotFound desc = could not find container \"4190d727f017290ade42507dda2560f8a07c5840fe01a98a4a44f595bfccfbb1\": container with ID starting with 4190d727f017290ade42507dda2560f8a07c5840fe01a98a4a44f595bfccfbb1 not found: ID does not exist" Jul 22 17:10:03.055879 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:03.055847 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr"] Jul 22 17:10:03.064552 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:03.064529 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-af56de08-656bdc44c8-xxmgr"] Jul 22 17:10:03.099863 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:03.099835 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnjdz\" (UniqueName: \"kubernetes.io/projected/fa278095-559c-413b-85c4-dff17ec6267e-kube-api-access-lnjdz\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:10:03.099863 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:03.099859 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa278095-559c-413b-85c4-dff17ec6267e-maas-api-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:10:03.099863 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:03.099871 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa278095-559c-413b-85c4-dff17ec6267e-openshift-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:10:03.574751 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:03.574718 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa278095-559c-413b-85c4-dff17ec6267e" path="/var/lib/kubelet/pods/fa278095-559c-413b-85c4-dff17ec6267e/volumes" Jul 22 17:10:07.188864 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:07.188833 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw"] Jul 22 17:10:07.191491 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:07.191464 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-af56de08-x9jkw"] Jul 22 17:10:07.294667 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:07.294631 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:07.417732 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:07.417697 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:07.574516 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:07.574484 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1ff9250-365c-4151-bc45-a4deb625d020" path="/var/lib/kubelet/pods/f1ff9250-365c-4151-bc45-a4deb625d020/volumes" Jul 22 17:10:07.623883 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:07.623843 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:07.832525 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:07.832417 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:08.145823 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:08.145739 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:08.480982 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:08.480946 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:10.159725 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.159684 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:10.272223 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.272190 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:10.340428 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.340396 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv"] Jul 22 17:10:10.340787 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.340775 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1ff9250-365c-4151-bc45-a4deb625d020" containerName="revoke-keys" Jul 22 17:10:10.340787 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.340787 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1ff9250-365c-4151-bc45-a4deb625d020" containerName="revoke-keys" Jul 22 17:10:10.340969 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.340801 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa278095-559c-413b-85c4-dff17ec6267e" containerName="maas-api" Jul 22 17:10:10.340969 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.340807 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa278095-559c-413b-85c4-dff17ec6267e" containerName="maas-api" Jul 22 17:10:10.340969 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.340874 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1ff9250-365c-4151-bc45-a4deb625d020" containerName="revoke-keys" Jul 22 17:10:10.340969 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.340885 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa278095-559c-413b-85c4-dff17ec6267e" containerName="maas-api" Jul 22 17:10:10.344093 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.344071 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:10.348187 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.348169 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b408d9d9-serving-cert\"" Jul 22 17:10:10.348599 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.348582 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8sp7j\"" Jul 22 17:10:10.363615 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.363590 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv"] Jul 22 17:10:10.473446 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.473351 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3ef3fc-7133-4932-901f-a3e005865b02-maas-api-tls\") pod \"maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv\" (UID: \"4e3ef3fc-7133-4932-901f-a3e005865b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:10.473446 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.473415 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4xdt\" (UniqueName: \"kubernetes.io/projected/4e3ef3fc-7133-4932-901f-a3e005865b02-kube-api-access-r4xdt\") pod \"maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv\" (UID: \"4e3ef3fc-7133-4932-901f-a3e005865b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:10.473644 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.473491 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3ef3fc-7133-4932-901f-a3e005865b02-openshift-service-ca\") pod \"maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv\" (UID: \"4e3ef3fc-7133-4932-901f-a3e005865b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:10.541121 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.541093 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:10.574444 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.574418 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3ef3fc-7133-4932-901f-a3e005865b02-maas-api-tls\") pod \"maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv\" (UID: \"4e3ef3fc-7133-4932-901f-a3e005865b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:10.574560 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.574465 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4xdt\" (UniqueName: \"kubernetes.io/projected/4e3ef3fc-7133-4932-901f-a3e005865b02-kube-api-access-r4xdt\") pod \"maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv\" (UID: \"4e3ef3fc-7133-4932-901f-a3e005865b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:10.574560 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.574504 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3ef3fc-7133-4932-901f-a3e005865b02-openshift-service-ca\") pod \"maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv\" (UID: \"4e3ef3fc-7133-4932-901f-a3e005865b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:10.574633 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:10:10.574562 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-serving-cert: secret "maas-api-e2e-ait-b408d9d9-serving-cert" not found Jul 22 17:10:10.574667 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:10:10.574640 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3ef3fc-7133-4932-901f-a3e005865b02-maas-api-tls podName:4e3ef3fc-7133-4932-901f-a3e005865b02 nodeName:}" failed. No retries permitted until 2026-07-22 17:10:11.074624904 +0000 UTC m=+1776.051353197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e3ef3fc-7133-4932-901f-a3e005865b02-maas-api-tls") pod "maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" (UID: "4e3ef3fc-7133-4932-901f-a3e005865b02") : secret "maas-api-e2e-ait-b408d9d9-serving-cert" not found Jul 22 17:10:10.575100 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.575084 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3ef3fc-7133-4932-901f-a3e005865b02-openshift-service-ca\") pod \"maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv\" (UID: \"4e3ef3fc-7133-4932-901f-a3e005865b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:10.589199 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.589172 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4xdt\" (UniqueName: \"kubernetes.io/projected/4e3ef3fc-7133-4932-901f-a3e005865b02-kube-api-access-r4xdt\") pod \"maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv\" (UID: \"4e3ef3fc-7133-4932-901f-a3e005865b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:10.714361 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:10.714325 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:11.078451 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:11.078418 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3ef3fc-7133-4932-901f-a3e005865b02-maas-api-tls\") pod \"maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv\" (UID: \"4e3ef3fc-7133-4932-901f-a3e005865b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:11.081097 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:11.081070 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3ef3fc-7133-4932-901f-a3e005865b02-maas-api-tls\") pod \"maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv\" (UID: \"4e3ef3fc-7133-4932-901f-a3e005865b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:11.149838 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:11.149798 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:11.253992 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:11.253957 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:11.272432 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:11.272391 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:11.397792 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:11.397766 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv"] Jul 22 17:10:11.400588 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:10:11.400561 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3ef3fc_7133_4932_901f_a3e005865b02.slice/crio-8c59a67b4c15351a83a4cf7b367a044afb35f43e888f310dab6b63f525c7dfc6 WatchSource:0}: Error finding container 8c59a67b4c15351a83a4cf7b367a044afb35f43e888f310dab6b63f525c7dfc6: Status 404 returned error can't find the container with id 8c59a67b4c15351a83a4cf7b367a044afb35f43e888f310dab6b63f525c7dfc6 Jul 22 17:10:12.080866 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:12.080777 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" event={"ID":"4e3ef3fc-7133-4932-901f-a3e005865b02","Type":"ContainerStarted","Data":"cc63974ba2e1fac5fbe12b0c485d071333a7c6c809032c1b810767e7e758bfa7"} Jul 22 17:10:12.080866 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:12.080842 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" event={"ID":"4e3ef3fc-7133-4932-901f-a3e005865b02","Type":"ContainerStarted","Data":"8c59a67b4c15351a83a4cf7b367a044afb35f43e888f310dab6b63f525c7dfc6"} Jul 22 17:10:12.081082 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:12.080913 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:12.102003 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:12.101955 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" podStartSLOduration=1.726154571 podStartE2EDuration="2.10193892s" totalImagesPullingTime="375.784349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:10:12.101121441 +0000 UTC m=+1777.077849758" watchObservedRunningTime="2026-07-22 17:10:12.10193892 +0000 UTC m=+1777.078667226" Jul 22 17:10:18.091834 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:18.091804 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:33.021064 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:33.020971 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:35.939266 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:35.939191 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7"] Jul 22 17:10:35.943789 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:35.943769 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7" Jul 22 17:10:35.946442 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:35.946421 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vln7b\"" Jul 22 17:10:35.951251 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:35.951225 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7"] Jul 22 17:10:35.999889 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:35.999859 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f11e43f-4cba-4e14-89a5-4fe72b4a974f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7\" (UID: \"0f11e43f-4cba-4e14-89a5-4fe72b4a974f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7" Jul 22 17:10:36.100505 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:36.100470 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f11e43f-4cba-4e14-89a5-4fe72b4a974f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7\" (UID: \"0f11e43f-4cba-4e14-89a5-4fe72b4a974f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7" Jul 22 17:10:36.101100 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:36.101082 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f11e43f-4cba-4e14-89a5-4fe72b4a974f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7\" (UID: \"0f11e43f-4cba-4e14-89a5-4fe72b4a974f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7" Jul 22 17:10:36.255984 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:36.255953 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7" Jul 22 17:10:36.396337 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:36.396296 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7"] Jul 22 17:10:36.399809 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:10:36.399780 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f11e43f_4cba_4e14_89a5_4fe72b4a974f.slice/crio-e5083fb394f372ebbfc354f79b550b2ed06b7c2ffda38203bf99343455a6e6f4 WatchSource:0}: Error finding container e5083fb394f372ebbfc354f79b550b2ed06b7c2ffda38203bf99343455a6e6f4: Status 404 returned error can't find the container with id e5083fb394f372ebbfc354f79b550b2ed06b7c2ffda38203bf99343455a6e6f4 Jul 22 17:10:37.205817 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:37.205775 2557 generic.go:356] "Generic (PLEG): container finished" podID="0f11e43f-4cba-4e14-89a5-4fe72b4a974f" containerID="2c80e21525db1923870d383262f874db0bdc72627c233669a7faf61ce2e09592" exitCode=0 Jul 22 17:10:37.206268 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:37.205855 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7" event={"ID":"0f11e43f-4cba-4e14-89a5-4fe72b4a974f","Type":"ContainerDied","Data":"2c80e21525db1923870d383262f874db0bdc72627c233669a7faf61ce2e09592"} Jul 22 17:10:37.206268 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:37.205913 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7" event={"ID":"0f11e43f-4cba-4e14-89a5-4fe72b4a974f","Type":"ContainerStarted","Data":"e5083fb394f372ebbfc354f79b550b2ed06b7c2ffda38203bf99343455a6e6f4"} Jul 22 17:10:38.343181 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:38.343149 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7" Jul 22 17:10:38.420366 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:38.420333 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f11e43f-4cba-4e14-89a5-4fe72b4a974f-maas-api-service-ca\") pod \"0f11e43f-4cba-4e14-89a5-4fe72b4a974f\" (UID: \"0f11e43f-4cba-4e14-89a5-4fe72b4a974f\") " Jul 22 17:10:38.420720 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:38.420694 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f11e43f-4cba-4e14-89a5-4fe72b4a974f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f11e43f-4cba-4e14-89a5-4fe72b4a974f" (UID: "0f11e43f-4cba-4e14-89a5-4fe72b4a974f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:10:38.521969 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:38.521923 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f11e43f-4cba-4e14-89a5-4fe72b4a974f-maas-api-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:10:39.215866 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:39.215838 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7" Jul 22 17:10:39.216057 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:39.215878 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7" event={"ID":"0f11e43f-4cba-4e14-89a5-4fe72b4a974f","Type":"ContainerDied","Data":"e5083fb394f372ebbfc354f79b550b2ed06b7c2ffda38203bf99343455a6e6f4"} Jul 22 17:10:39.216057 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:39.215924 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5083fb394f372ebbfc354f79b550b2ed06b7c2ffda38203bf99343455a6e6f4" Jul 22 17:10:46.615745 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:46.615705 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv"] Jul 22 17:10:46.616224 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:46.615967 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" podUID="4e3ef3fc-7133-4932-901f-a3e005865b02" containerName="maas-api" containerID="cri-o://cc63974ba2e1fac5fbe12b0c485d071333a7c6c809032c1b810767e7e758bfa7" gracePeriod=30 Jul 22 17:10:46.654324 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:46.654290 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:46.883084 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:46.883056 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:46.998802 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:46.998758 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4xdt\" (UniqueName: \"kubernetes.io/projected/4e3ef3fc-7133-4932-901f-a3e005865b02-kube-api-access-r4xdt\") pod \"4e3ef3fc-7133-4932-901f-a3e005865b02\" (UID: \"4e3ef3fc-7133-4932-901f-a3e005865b02\") " Jul 22 17:10:46.998981 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:46.998815 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3ef3fc-7133-4932-901f-a3e005865b02-openshift-service-ca\") pod \"4e3ef3fc-7133-4932-901f-a3e005865b02\" (UID: \"4e3ef3fc-7133-4932-901f-a3e005865b02\") " Jul 22 17:10:46.998981 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:46.998837 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3ef3fc-7133-4932-901f-a3e005865b02-maas-api-tls\") pod \"4e3ef3fc-7133-4932-901f-a3e005865b02\" (UID: \"4e3ef3fc-7133-4932-901f-a3e005865b02\") " Jul 22 17:10:46.999246 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:46.999215 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3ef3fc-7133-4932-901f-a3e005865b02-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e3ef3fc-7133-4932-901f-a3e005865b02" (UID: "4e3ef3fc-7133-4932-901f-a3e005865b02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:10:47.001003 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:47.000972 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3ef3fc-7133-4932-901f-a3e005865b02-kube-api-access-r4xdt" (OuterVolumeSpecName: "kube-api-access-r4xdt") pod "4e3ef3fc-7133-4932-901f-a3e005865b02" (UID: "4e3ef3fc-7133-4932-901f-a3e005865b02"). InnerVolumeSpecName "kube-api-access-r4xdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:10:47.001151 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:47.001120 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e3ef3fc-7133-4932-901f-a3e005865b02-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e3ef3fc-7133-4932-901f-a3e005865b02" (UID: "4e3ef3fc-7133-4932-901f-a3e005865b02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:10:47.099789 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:47.099751 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4xdt\" (UniqueName: \"kubernetes.io/projected/4e3ef3fc-7133-4932-901f-a3e005865b02-kube-api-access-r4xdt\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:10:47.099789 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:47.099783 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3ef3fc-7133-4932-901f-a3e005865b02-openshift-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:10:47.099789 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:47.099792 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3ef3fc-7133-4932-901f-a3e005865b02-maas-api-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:10:47.259726 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:47.259687 2557 generic.go:356] "Generic (PLEG): container finished" podID="4e3ef3fc-7133-4932-901f-a3e005865b02" containerID="cc63974ba2e1fac5fbe12b0c485d071333a7c6c809032c1b810767e7e758bfa7" exitCode=0 Jul 22 17:10:47.259726 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:47.259730 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" Jul 22 17:10:47.259952 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:47.259775 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" event={"ID":"4e3ef3fc-7133-4932-901f-a3e005865b02","Type":"ContainerDied","Data":"cc63974ba2e1fac5fbe12b0c485d071333a7c6c809032c1b810767e7e758bfa7"} Jul 22 17:10:47.259952 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:47.259825 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv" event={"ID":"4e3ef3fc-7133-4932-901f-a3e005865b02","Type":"ContainerDied","Data":"8c59a67b4c15351a83a4cf7b367a044afb35f43e888f310dab6b63f525c7dfc6"} Jul 22 17:10:47.259952 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:47.259847 2557 scope.go:117] "RemoveContainer" containerID="cc63974ba2e1fac5fbe12b0c485d071333a7c6c809032c1b810767e7e758bfa7" Jul 22 17:10:47.269167 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:47.269146 2557 scope.go:117] "RemoveContainer" containerID="cc63974ba2e1fac5fbe12b0c485d071333a7c6c809032c1b810767e7e758bfa7" Jul 22 17:10:47.269516 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:10:47.269495 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc63974ba2e1fac5fbe12b0c485d071333a7c6c809032c1b810767e7e758bfa7\": container with ID starting with cc63974ba2e1fac5fbe12b0c485d071333a7c6c809032c1b810767e7e758bfa7 not found: ID does not exist" containerID="cc63974ba2e1fac5fbe12b0c485d071333a7c6c809032c1b810767e7e758bfa7" Jul 22 17:10:47.269621 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:47.269522 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc63974ba2e1fac5fbe12b0c485d071333a7c6c809032c1b810767e7e758bfa7"} err="failed to get container status \"cc63974ba2e1fac5fbe12b0c485d071333a7c6c809032c1b810767e7e758bfa7\": rpc error: code = NotFound desc = could not find container \"cc63974ba2e1fac5fbe12b0c485d071333a7c6c809032c1b810767e7e758bfa7\": container with ID starting with cc63974ba2e1fac5fbe12b0c485d071333a7c6c809032c1b810767e7e758bfa7 not found: ID does not exist" Jul 22 17:10:47.281999 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:47.281970 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv"] Jul 22 17:10:47.285839 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:47.285810 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b408d9d9-698f9b8444-rddlv"] Jul 22 17:10:47.575723 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:47.575630 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e3ef3fc-7133-4932-901f-a3e005865b02" path="/var/lib/kubelet/pods/4e3ef3fc-7133-4932-901f-a3e005865b02/volumes" Jul 22 17:10:51.024489 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:51.024451 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7"] Jul 22 17:10:51.030706 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:51.030673 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b408d9d9-gm7x7"] Jul 22 17:10:51.052314 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:51.052282 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:51.575541 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:51.575498 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f11e43f-4cba-4e14-89a5-4fe72b4a974f" path="/var/lib/kubelet/pods/0f11e43f-4cba-4e14-89a5-4fe72b4a974f/volumes" Jul 22 17:10:52.290314 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:52.290277 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:52.534620 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:52.534585 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:52.779598 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:52.779565 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:53.137224 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:53.137125 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:53.243935 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:53.243898 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:55.377148 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.377111 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:55.436758 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.436722 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s"] Jul 22 17:10:55.437550 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.437526 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e3ef3fc-7133-4932-901f-a3e005865b02" containerName="maas-api" Jul 22 17:10:55.437695 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.437580 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e3ef3fc-7133-4932-901f-a3e005865b02" containerName="maas-api" Jul 22 17:10:55.437695 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.437625 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f11e43f-4cba-4e14-89a5-4fe72b4a974f" containerName="revoke-keys" Jul 22 17:10:55.437695 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.437634 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f11e43f-4cba-4e14-89a5-4fe72b4a974f" containerName="revoke-keys" Jul 22 17:10:55.437856 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.437735 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f11e43f-4cba-4e14-89a5-4fe72b4a974f" containerName="revoke-keys" Jul 22 17:10:55.437856 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.437755 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e3ef3fc-7133-4932-901f-a3e005865b02" containerName="maas-api" Jul 22 17:10:55.442170 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.442150 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" Jul 22 17:10:55.444910 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.444892 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f443994a-serving-cert\"" Jul 22 17:10:55.445063 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.444898 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8sp7j\"" Jul 22 17:10:55.453718 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.453691 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s"] Jul 22 17:10:55.502927 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.502896 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:55.578735 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.578707 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf1628a-cd4c-45ba-ba0a-21629648bd96-maas-api-tls\") pod \"maas-api-e2e-derive-f443994a-99c786849-8726s\" (UID: \"dcf1628a-cd4c-45ba-ba0a-21629648bd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" Jul 22 17:10:55.578908 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.578748 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql8hk\" (UniqueName: \"kubernetes.io/projected/dcf1628a-cd4c-45ba-ba0a-21629648bd96-kube-api-access-ql8hk\") pod \"maas-api-e2e-derive-f443994a-99c786849-8726s\" (UID: \"dcf1628a-cd4c-45ba-ba0a-21629648bd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" Jul 22 17:10:55.578908 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.578854 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf1628a-cd4c-45ba-ba0a-21629648bd96-openshift-service-ca\") pod \"maas-api-e2e-derive-f443994a-99c786849-8726s\" (UID: \"dcf1628a-cd4c-45ba-ba0a-21629648bd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" Jul 22 17:10:55.601439 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.601408 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:55.679427 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.679332 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf1628a-cd4c-45ba-ba0a-21629648bd96-maas-api-tls\") pod \"maas-api-e2e-derive-f443994a-99c786849-8726s\" (UID: \"dcf1628a-cd4c-45ba-ba0a-21629648bd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" Jul 22 17:10:55.679427 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.679392 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql8hk\" (UniqueName: \"kubernetes.io/projected/dcf1628a-cd4c-45ba-ba0a-21629648bd96-kube-api-access-ql8hk\") pod \"maas-api-e2e-derive-f443994a-99c786849-8726s\" (UID: \"dcf1628a-cd4c-45ba-ba0a-21629648bd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" Jul 22 17:10:55.679732 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.679456 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf1628a-cd4c-45ba-ba0a-21629648bd96-openshift-service-ca\") pod \"maas-api-e2e-derive-f443994a-99c786849-8726s\" (UID: \"dcf1628a-cd4c-45ba-ba0a-21629648bd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" Jul 22 17:10:55.680136 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.680117 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf1628a-cd4c-45ba-ba0a-21629648bd96-openshift-service-ca\") pod \"maas-api-e2e-derive-f443994a-99c786849-8726s\" (UID: \"dcf1628a-cd4c-45ba-ba0a-21629648bd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" Jul 22 17:10:55.681722 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.681698 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf1628a-cd4c-45ba-ba0a-21629648bd96-maas-api-tls\") pod \"maas-api-e2e-derive-f443994a-99c786849-8726s\" (UID: \"dcf1628a-cd4c-45ba-ba0a-21629648bd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" Jul 22 17:10:55.688742 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.688715 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql8hk\" (UniqueName: \"kubernetes.io/projected/dcf1628a-cd4c-45ba-ba0a-21629648bd96-kube-api-access-ql8hk\") pod \"maas-api-e2e-derive-f443994a-99c786849-8726s\" (UID: \"dcf1628a-cd4c-45ba-ba0a-21629648bd96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" Jul 22 17:10:55.754161 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.754133 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" Jul 22 17:10:55.891538 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:55.891501 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s"] Jul 22 17:10:55.894544 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:10:55.894505 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf1628a_cd4c_45ba_ba0a_21629648bd96.slice/crio-d2ac54ba7984c7ffb12b0ade74175350e6b2c343b2eb75b47f82df163a6d8027 WatchSource:0}: Error finding container d2ac54ba7984c7ffb12b0ade74175350e6b2c343b2eb75b47f82df163a6d8027: Status 404 returned error can't find the container with id d2ac54ba7984c7ffb12b0ade74175350e6b2c343b2eb75b47f82df163a6d8027 Jul 22 17:10:56.104586 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:56.104546 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:56.253867 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:56.253825 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:10:56.309452 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:56.309417 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" event={"ID":"dcf1628a-cd4c-45ba-ba0a-21629648bd96","Type":"ContainerStarted","Data":"d2ac54ba7984c7ffb12b0ade74175350e6b2c343b2eb75b47f82df163a6d8027"} Jul 22 17:10:57.315518 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:57.315481 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" event={"ID":"dcf1628a-cd4c-45ba-ba0a-21629648bd96","Type":"ContainerStarted","Data":"55adec1b53277e87c744b1876ad6ceb3ffaeb74b52389275ceff7a5278a0f550"} Jul 22 17:10:57.315951 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:57.315610 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" Jul 22 17:10:57.338143 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:10:57.338093 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" podStartSLOduration=1.8734881410000002 podStartE2EDuration="2.338077053s" totalImagesPullingTime="464.588912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:10:57.336556576 +0000 UTC m=+1822.313285021" watchObservedRunningTime="2026-07-22 17:10:57.338077053 +0000 UTC m=+1822.314805366" Jul 22 17:11:03.325793 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:03.325762 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" Jul 22 17:11:09.504009 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:09.503965 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:09.662082 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:09.662045 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:17.287773 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:17.287739 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:17.410791 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:17.410756 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:20.211302 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:20.211264 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f443994a-7m8rm"] Jul 22 17:11:20.214391 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:20.214374 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f443994a-7m8rm" Jul 22 17:11:20.217140 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:20.217116 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vln7b\"" Jul 22 17:11:20.223910 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:20.223887 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f443994a-7m8rm"] Jul 22 17:11:20.316641 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:20.316605 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934a9fca-6405-4d4e-aef7-6e17da525f1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f443994a-7m8rm\" (UID: \"934a9fca-6405-4d4e-aef7-6e17da525f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f443994a-7m8rm" Jul 22 17:11:20.418188 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:20.418148 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934a9fca-6405-4d4e-aef7-6e17da525f1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f443994a-7m8rm\" (UID: \"934a9fca-6405-4d4e-aef7-6e17da525f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f443994a-7m8rm" Jul 22 17:11:20.418794 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:20.418774 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934a9fca-6405-4d4e-aef7-6e17da525f1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f443994a-7m8rm\" (UID: \"934a9fca-6405-4d4e-aef7-6e17da525f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f443994a-7m8rm" Jul 22 17:11:20.526375 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:20.526342 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f443994a-7m8rm" Jul 22 17:11:20.668053 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:20.668014 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f443994a-7m8rm"] Jul 22 17:11:20.670396 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:11:20.670368 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934a9fca_6405_4d4e_aef7_6e17da525f1a.slice/crio-15984d072e00527f8966a689e219ff5d76d7c3ffc37582f4aba9ee2d574261b1 WatchSource:0}: Error finding container 15984d072e00527f8966a689e219ff5d76d7c3ffc37582f4aba9ee2d574261b1: Status 404 returned error can't find the container with id 15984d072e00527f8966a689e219ff5d76d7c3ffc37582f4aba9ee2d574261b1 Jul 22 17:11:21.393283 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:11:21.393255 2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934a9fca_6405_4d4e_aef7_6e17da525f1a.slice/crio-conmon-c67f03a3ca27a7bff786de6a0846ef6e52c890559b07f5a13428fb082f737d26.scope\": RecentStats: unable to find data in memory cache]" Jul 22 17:11:21.441124 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:21.441091 2557 generic.go:356] "Generic (PLEG): container finished" podID="934a9fca-6405-4d4e-aef7-6e17da525f1a" containerID="c67f03a3ca27a7bff786de6a0846ef6e52c890559b07f5a13428fb082f737d26" exitCode=0 Jul 22 17:11:21.441301 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:21.441138 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f443994a-7m8rm" event={"ID":"934a9fca-6405-4d4e-aef7-6e17da525f1a","Type":"ContainerDied","Data":"c67f03a3ca27a7bff786de6a0846ef6e52c890559b07f5a13428fb082f737d26"} Jul 22 17:11:21.441301 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:21.441175 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f443994a-7m8rm" event={"ID":"934a9fca-6405-4d4e-aef7-6e17da525f1a","Type":"ContainerStarted","Data":"15984d072e00527f8966a689e219ff5d76d7c3ffc37582f4aba9ee2d574261b1"} Jul 22 17:11:22.573231 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:22.573205 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f443994a-7m8rm" Jul 22 17:11:22.638543 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:22.638515 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934a9fca-6405-4d4e-aef7-6e17da525f1a-maas-api-service-ca\") pod \"934a9fca-6405-4d4e-aef7-6e17da525f1a\" (UID: \"934a9fca-6405-4d4e-aef7-6e17da525f1a\") " Jul 22 17:11:22.638914 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:22.638887 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934a9fca-6405-4d4e-aef7-6e17da525f1a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "934a9fca-6405-4d4e-aef7-6e17da525f1a" (UID: "934a9fca-6405-4d4e-aef7-6e17da525f1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:11:22.739471 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:22.739443 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934a9fca-6405-4d4e-aef7-6e17da525f1a-maas-api-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:11:23.452915 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:23.452880 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f443994a-7m8rm" Jul 22 17:11:23.453109 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:23.452879 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f443994a-7m8rm" event={"ID":"934a9fca-6405-4d4e-aef7-6e17da525f1a","Type":"ContainerDied","Data":"15984d072e00527f8966a689e219ff5d76d7c3ffc37582f4aba9ee2d574261b1"} Jul 22 17:11:23.453109 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:23.452996 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15984d072e00527f8966a689e219ff5d76d7c3ffc37582f4aba9ee2d574261b1" Jul 22 17:11:30.756378 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:30.756344 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s"] Jul 22 17:11:30.756818 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:30.756547 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" podUID="dcf1628a-cd4c-45ba-ba0a-21629648bd96" containerName="maas-api" containerID="cri-o://55adec1b53277e87c744b1876ad6ceb3ffaeb74b52389275ceff7a5278a0f550" gracePeriod=30 Jul 22 17:11:30.776147 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:30.776118 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:31.007565 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.007506 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" Jul 22 17:11:31.117345 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.117309 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf1628a-cd4c-45ba-ba0a-21629648bd96-openshift-service-ca\") pod \"dcf1628a-cd4c-45ba-ba0a-21629648bd96\" (UID: \"dcf1628a-cd4c-45ba-ba0a-21629648bd96\") " Jul 22 17:11:31.117518 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.117373 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql8hk\" (UniqueName: \"kubernetes.io/projected/dcf1628a-cd4c-45ba-ba0a-21629648bd96-kube-api-access-ql8hk\") pod \"dcf1628a-cd4c-45ba-ba0a-21629648bd96\" (UID: \"dcf1628a-cd4c-45ba-ba0a-21629648bd96\") " Jul 22 17:11:31.117518 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.117506 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf1628a-cd4c-45ba-ba0a-21629648bd96-maas-api-tls\") pod \"dcf1628a-cd4c-45ba-ba0a-21629648bd96\" (UID: \"dcf1628a-cd4c-45ba-ba0a-21629648bd96\") " Jul 22 17:11:31.117709 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.117683 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf1628a-cd4c-45ba-ba0a-21629648bd96-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dcf1628a-cd4c-45ba-ba0a-21629648bd96" (UID: "dcf1628a-cd4c-45ba-ba0a-21629648bd96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:11:31.117766 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.117753 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf1628a-cd4c-45ba-ba0a-21629648bd96-openshift-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:11:31.119497 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.119474 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf1628a-cd4c-45ba-ba0a-21629648bd96-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dcf1628a-cd4c-45ba-ba0a-21629648bd96" (UID: "dcf1628a-cd4c-45ba-ba0a-21629648bd96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:11:31.119583 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.119520 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf1628a-cd4c-45ba-ba0a-21629648bd96-kube-api-access-ql8hk" (OuterVolumeSpecName: "kube-api-access-ql8hk") pod "dcf1628a-cd4c-45ba-ba0a-21629648bd96" (UID: "dcf1628a-cd4c-45ba-ba0a-21629648bd96"). InnerVolumeSpecName "kube-api-access-ql8hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:11:31.219067 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.219019 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql8hk\" (UniqueName: \"kubernetes.io/projected/dcf1628a-cd4c-45ba-ba0a-21629648bd96-kube-api-access-ql8hk\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:11:31.219067 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.219064 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf1628a-cd4c-45ba-ba0a-21629648bd96-maas-api-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:11:31.493463 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.493427 2557 generic.go:356] "Generic (PLEG): container finished" podID="dcf1628a-cd4c-45ba-ba0a-21629648bd96" containerID="55adec1b53277e87c744b1876ad6ceb3ffaeb74b52389275ceff7a5278a0f550" exitCode=0 Jul 22 17:11:31.493635 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.493469 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" Jul 22 17:11:31.493635 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.493510 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" event={"ID":"dcf1628a-cd4c-45ba-ba0a-21629648bd96","Type":"ContainerDied","Data":"55adec1b53277e87c744b1876ad6ceb3ffaeb74b52389275ceff7a5278a0f550"} Jul 22 17:11:31.493635 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.493566 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s" event={"ID":"dcf1628a-cd4c-45ba-ba0a-21629648bd96","Type":"ContainerDied","Data":"d2ac54ba7984c7ffb12b0ade74175350e6b2c343b2eb75b47f82df163a6d8027"} Jul 22 17:11:31.493635 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.493594 2557 scope.go:117] "RemoveContainer" containerID="55adec1b53277e87c744b1876ad6ceb3ffaeb74b52389275ceff7a5278a0f550" Jul 22 17:11:31.502802 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.502785 2557 scope.go:117] "RemoveContainer" containerID="55adec1b53277e87c744b1876ad6ceb3ffaeb74b52389275ceff7a5278a0f550" Jul 22 17:11:31.503084 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:11:31.503064 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55adec1b53277e87c744b1876ad6ceb3ffaeb74b52389275ceff7a5278a0f550\": container with ID starting with 55adec1b53277e87c744b1876ad6ceb3ffaeb74b52389275ceff7a5278a0f550 not found: ID does not exist" containerID="55adec1b53277e87c744b1876ad6ceb3ffaeb74b52389275ceff7a5278a0f550" Jul 22 17:11:31.503170 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.503090 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55adec1b53277e87c744b1876ad6ceb3ffaeb74b52389275ceff7a5278a0f550"} err="failed to get container status \"55adec1b53277e87c744b1876ad6ceb3ffaeb74b52389275ceff7a5278a0f550\": rpc error: code = NotFound desc = could not find container \"55adec1b53277e87c744b1876ad6ceb3ffaeb74b52389275ceff7a5278a0f550\": container with ID starting with 55adec1b53277e87c744b1876ad6ceb3ffaeb74b52389275ceff7a5278a0f550 not found: ID does not exist" Jul 22 17:11:31.515920 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.515890 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s"] Jul 22 17:11:31.520175 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.520155 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f443994a-99c786849-8726s"] Jul 22 17:11:31.576107 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:31.576083 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcf1628a-cd4c-45ba-ba0a-21629648bd96" path="/var/lib/kubelet/pods/dcf1628a-cd4c-45ba-ba0a-21629648bd96/volumes" Jul 22 17:11:35.305707 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:35.305667 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f443994a-7m8rm"] Jul 22 17:11:35.309973 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:35.309943 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:35.310192 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:35.310174 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f443994a-7m8rm"] Jul 22 17:11:35.461682 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:35.461648 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:35.574702 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:35.574628 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="934a9fca-6405-4d4e-aef7-6e17da525f1a" path="/var/lib/kubelet/pods/934a9fca-6405-4d4e-aef7-6e17da525f1a/volumes" Jul 22 17:11:35.701833 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:35.701801 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:36.198257 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:36.198224 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:36.290297 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:36.290264 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:36.391574 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:36.391545 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:37.967903 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:37.967866 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:44.255727 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:44.255694 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:44.315132 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:44.315097 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:44.410254 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:44.410217 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:44.502661 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:44.502627 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:44.609936 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:44.609851 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:45.061683 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:45.061654 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:49.039065 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:49.039002 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:11:49.111444 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:11:49.111409 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:12:11.047505 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:12:11.047437 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:12:18.755253 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:12:18.755221 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:12:18.901599 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:12:18.901561 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:12:19.024718 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:12:19.024634 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:12:34.902143 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:12:34.902108 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:12:41.301391 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:12:41.301348 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:12:41.359939 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:12:41.359904 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:12:41.477421 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:12:41.477389 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:12:41.575362 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:12:41.575272 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:12:52.098945 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:12:52.098906 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:12:52.927182 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:12:52.927150 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:12:59.918395 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:12:59.918355 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:00.023383 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:00.023336 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:00.121878 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:00.121849 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:00.225209 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:00.225131 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:01.082642 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:01.082604 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:01.178171 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:01.178134 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:01.275951 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:01.275914 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:02.924325 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:02.924294 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:12.279225 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:12.279190 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:13.554565 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:13.554522 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:17.595190 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:17.595155 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:20.000669 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:20.000635 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:20.105847 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:20.105813 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:20.256826 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:20.256743 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:20.695353 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:20.695265 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:21.027776 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:21.027737 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:23.843388 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:23.843318 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:23.940703 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:23.940669 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:23.983787 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:23.983757 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z"] Jul 22 17:13:23.984223 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:23.984195 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf1628a-cd4c-45ba-ba0a-21629648bd96" containerName="maas-api" Jul 22 17:13:23.984223 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:23.984216 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf1628a-cd4c-45ba-ba0a-21629648bd96" containerName="maas-api" Jul 22 17:13:23.984424 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:23.984243 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="934a9fca-6405-4d4e-aef7-6e17da525f1a" containerName="revoke-keys" Jul 22 17:13:23.984424 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:23.984249 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="934a9fca-6405-4d4e-aef7-6e17da525f1a" containerName="revoke-keys" Jul 22 17:13:23.984424 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:23.984319 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf1628a-cd4c-45ba-ba0a-21629648bd96" containerName="maas-api" Jul 22 17:13:23.984424 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:23.984328 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="934a9fca-6405-4d4e-aef7-6e17da525f1a" containerName="revoke-keys" Jul 22 17:13:23.987509 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:23.987491 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" Jul 22 17:13:23.990581 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:23.990557 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8sp7j\"" Jul 22 17:13:23.990934 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:23.990917 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-51445c0e-serving-cert\"" Jul 22 17:13:24.054290 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.054259 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z"] Jul 22 17:13:24.066181 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.066152 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzw2z\" (UniqueName: \"kubernetes.io/projected/21a2f9bb-259e-4cd8-a738-36e9943aad75-kube-api-access-pzw2z\") pod \"maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z\" (UID: \"21a2f9bb-259e-4cd8-a738-36e9943aad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" Jul 22 17:13:24.066338 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.066285 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a2f9bb-259e-4cd8-a738-36e9943aad75-maas-api-tls\") pod \"maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z\" (UID: \"21a2f9bb-259e-4cd8-a738-36e9943aad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" Jul 22 17:13:24.066338 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.066328 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a2f9bb-259e-4cd8-a738-36e9943aad75-openshift-service-ca\") pod \"maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z\" (UID: \"21a2f9bb-259e-4cd8-a738-36e9943aad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" Jul 22 17:13:24.069147 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.069108 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:24.167670 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.167574 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a2f9bb-259e-4cd8-a738-36e9943aad75-maas-api-tls\") pod \"maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z\" (UID: \"21a2f9bb-259e-4cd8-a738-36e9943aad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" Jul 22 17:13:24.167670 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.167618 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a2f9bb-259e-4cd8-a738-36e9943aad75-openshift-service-ca\") pod \"maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z\" (UID: \"21a2f9bb-259e-4cd8-a738-36e9943aad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" Jul 22 17:13:24.167885 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.167705 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzw2z\" (UniqueName: \"kubernetes.io/projected/21a2f9bb-259e-4cd8-a738-36e9943aad75-kube-api-access-pzw2z\") pod \"maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z\" (UID: \"21a2f9bb-259e-4cd8-a738-36e9943aad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" Jul 22 17:13:24.168341 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.168322 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a2f9bb-259e-4cd8-a738-36e9943aad75-openshift-service-ca\") pod \"maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z\" (UID: \"21a2f9bb-259e-4cd8-a738-36e9943aad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" Jul 22 17:13:24.170189 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.170159 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a2f9bb-259e-4cd8-a738-36e9943aad75-maas-api-tls\") pod \"maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z\" (UID: \"21a2f9bb-259e-4cd8-a738-36e9943aad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" Jul 22 17:13:24.177181 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.177153 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzw2z\" (UniqueName: \"kubernetes.io/projected/21a2f9bb-259e-4cd8-a738-36e9943aad75-kube-api-access-pzw2z\") pod \"maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z\" (UID: \"21a2f9bb-259e-4cd8-a738-36e9943aad75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" Jul 22 17:13:24.214651 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.214626 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:24.299012 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.298977 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" Jul 22 17:13:24.655683 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.655657 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z"] Jul 22 17:13:24.657666 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:13:24.657636 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a2f9bb_259e_4cd8_a738_36e9943aad75.slice/crio-b03e1bf971d1b683d759d7de8fec66b52557a8420d8648c6539b7715fc563480 WatchSource:0}: Error finding container b03e1bf971d1b683d759d7de8fec66b52557a8420d8648c6539b7715fc563480: Status 404 returned error can't find the container with id b03e1bf971d1b683d759d7de8fec66b52557a8420d8648c6539b7715fc563480 Jul 22 17:13:24.807367 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.807327 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:24.911235 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:24.911135 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:25.055391 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:25.055356 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" event={"ID":"21a2f9bb-259e-4cd8-a738-36e9943aad75","Type":"ContainerStarted","Data":"b03e1bf971d1b683d759d7de8fec66b52557a8420d8648c6539b7715fc563480"} Jul 22 17:13:26.061763 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:26.061730 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" event={"ID":"21a2f9bb-259e-4cd8-a738-36e9943aad75","Type":"ContainerStarted","Data":"e18b60bfe56df15698fa7f6182a479acb4a3dea7ad92e4d4fb3ac00e0b66bfb6"} Jul 22 17:13:26.062270 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:26.061859 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" Jul 22 17:13:26.095743 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:26.095695 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" podStartSLOduration=2.675282887 podStartE2EDuration="3.095680063s" totalImagesPullingTime="420.397176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:13:26.09513747 +0000 UTC m=+1971.071865788" watchObservedRunningTime="2026-07-22 17:13:26.095680063 +0000 UTC m=+1971.072408382" Jul 22 17:13:28.311237 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:28.311135 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:29.089446 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:29.089407 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:29.215576 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:29.215536 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:29.497729 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:29.497681 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:32.072303 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:32.072272 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" Jul 22 17:13:35.662652 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:35.662617 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:35.772276 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:35.772242 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:36.068706 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:36.068673 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw"] Jul 22 17:13:36.072439 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:36.072419 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw" Jul 22 17:13:36.075572 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:36.075554 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vln7b\"" Jul 22 17:13:36.102545 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:36.102515 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw"] Jul 22 17:13:36.177869 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:36.177834 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79653469-2dc3-4061-844b-a69a1481c2ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-51445c0e-d96qw\" (UID: \"79653469-2dc3-4061-844b-a69a1481c2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw" Jul 22 17:13:36.279288 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:36.279249 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79653469-2dc3-4061-844b-a69a1481c2ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-51445c0e-d96qw\" (UID: \"79653469-2dc3-4061-844b-a69a1481c2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw" Jul 22 17:13:36.279866 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:36.279848 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79653469-2dc3-4061-844b-a69a1481c2ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-51445c0e-d96qw\" (UID: \"79653469-2dc3-4061-844b-a69a1481c2ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw" Jul 22 17:13:36.382672 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:36.382600 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw" Jul 22 17:13:36.528611 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:36.528578 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw"] Jul 22 17:13:36.530686 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:13:36.530660 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79653469_2dc3_4061_844b_a69a1481c2ee.slice/crio-e95c0e5d6c8f7f8ac052b0f1e360523e524236b91708dd80b23bba81dfd192cd WatchSource:0}: Error finding container e95c0e5d6c8f7f8ac052b0f1e360523e524236b91708dd80b23bba81dfd192cd: Status 404 returned error can't find the container with id e95c0e5d6c8f7f8ac052b0f1e360523e524236b91708dd80b23bba81dfd192cd Jul 22 17:13:37.118178 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:37.118087 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw" event={"ID":"79653469-2dc3-4061-844b-a69a1481c2ee","Type":"ContainerStarted","Data":"b99e7b09e1e9e84ea1fc99acb1502eb8d5fe39e64b899d0dce6aa483a8e083bf"} Jul 22 17:13:37.118178 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:37.118145 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw" event={"ID":"79653469-2dc3-4061-844b-a69a1481c2ee","Type":"ContainerStarted","Data":"e95c0e5d6c8f7f8ac052b0f1e360523e524236b91708dd80b23bba81dfd192cd"} Jul 22 17:13:37.136804 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:37.136752 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw" podStartSLOduration=1.136732877 podStartE2EDuration="1.136732877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:13:37.136365689 +0000 UTC m=+1982.113094008" watchObservedRunningTime="2026-07-22 17:13:37.136732877 +0000 UTC m=+1982.113461220" Jul 22 17:13:38.125038 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:38.125000 2557 generic.go:356] "Generic (PLEG): container finished" podID="79653469-2dc3-4061-844b-a69a1481c2ee" containerID="b99e7b09e1e9e84ea1fc99acb1502eb8d5fe39e64b899d0dce6aa483a8e083bf" exitCode=0 Jul 22 17:13:38.125413 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:38.125084 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw" event={"ID":"79653469-2dc3-4061-844b-a69a1481c2ee","Type":"ContainerDied","Data":"b99e7b09e1e9e84ea1fc99acb1502eb8d5fe39e64b899d0dce6aa483a8e083bf"} Jul 22 17:13:39.263547 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:39.263522 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw" Jul 22 17:13:39.408606 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:39.408527 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79653469-2dc3-4061-844b-a69a1481c2ee-maas-api-service-ca\") pod \"79653469-2dc3-4061-844b-a69a1481c2ee\" (UID: \"79653469-2dc3-4061-844b-a69a1481c2ee\") " Jul 22 17:13:39.408866 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:39.408846 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79653469-2dc3-4061-844b-a69a1481c2ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "79653469-2dc3-4061-844b-a69a1481c2ee" (UID: "79653469-2dc3-4061-844b-a69a1481c2ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:13:39.509485 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:39.509454 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79653469-2dc3-4061-844b-a69a1481c2ee-maas-api-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:13:40.136222 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:40.136185 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw" event={"ID":"79653469-2dc3-4061-844b-a69a1481c2ee","Type":"ContainerDied","Data":"e95c0e5d6c8f7f8ac052b0f1e360523e524236b91708dd80b23bba81dfd192cd"} Jul 22 17:13:40.136222 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:40.136208 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw" Jul 22 17:13:40.136442 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:40.136240 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e95c0e5d6c8f7f8ac052b0f1e360523e524236b91708dd80b23bba81dfd192cd" Jul 22 17:13:46.676098 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:46.676067 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z"] Jul 22 17:13:46.676588 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:46.676280 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" podUID="21a2f9bb-259e-4cd8-a738-36e9943aad75" containerName="maas-api" containerID="cri-o://e18b60bfe56df15698fa7f6182a479acb4a3dea7ad92e4d4fb3ac00e0b66bfb6" gracePeriod=30 Jul 22 17:13:46.713131 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:46.713085 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:46.922515 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:46.922490 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" Jul 22 17:13:46.976221 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:46.976197 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a2f9bb-259e-4cd8-a738-36e9943aad75-maas-api-tls\") pod \"21a2f9bb-259e-4cd8-a738-36e9943aad75\" (UID: \"21a2f9bb-259e-4cd8-a738-36e9943aad75\") " Jul 22 17:13:46.976365 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:46.976236 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a2f9bb-259e-4cd8-a738-36e9943aad75-openshift-service-ca\") pod \"21a2f9bb-259e-4cd8-a738-36e9943aad75\" (UID: \"21a2f9bb-259e-4cd8-a738-36e9943aad75\") " Jul 22 17:13:46.976365 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:46.976269 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzw2z\" (UniqueName: \"kubernetes.io/projected/21a2f9bb-259e-4cd8-a738-36e9943aad75-kube-api-access-pzw2z\") pod \"21a2f9bb-259e-4cd8-a738-36e9943aad75\" (UID: \"21a2f9bb-259e-4cd8-a738-36e9943aad75\") " Jul 22 17:13:46.976644 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:46.976619 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a2f9bb-259e-4cd8-a738-36e9943aad75-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "21a2f9bb-259e-4cd8-a738-36e9943aad75" (UID: "21a2f9bb-259e-4cd8-a738-36e9943aad75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:13:46.978331 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:46.978300 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a2f9bb-259e-4cd8-a738-36e9943aad75-kube-api-access-pzw2z" (OuterVolumeSpecName: "kube-api-access-pzw2z") pod "21a2f9bb-259e-4cd8-a738-36e9943aad75" (UID: "21a2f9bb-259e-4cd8-a738-36e9943aad75"). InnerVolumeSpecName "kube-api-access-pzw2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:13:46.978524 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:46.978503 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a2f9bb-259e-4cd8-a738-36e9943aad75-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "21a2f9bb-259e-4cd8-a738-36e9943aad75" (UID: "21a2f9bb-259e-4cd8-a738-36e9943aad75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:13:47.077002 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:47.076964 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a2f9bb-259e-4cd8-a738-36e9943aad75-maas-api-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:13:47.077002 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:47.076996 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a2f9bb-259e-4cd8-a738-36e9943aad75-openshift-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:13:47.077002 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:47.077010 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzw2z\" (UniqueName: \"kubernetes.io/projected/21a2f9bb-259e-4cd8-a738-36e9943aad75-kube-api-access-pzw2z\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:13:47.174500 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:47.174461 2557 generic.go:356] "Generic (PLEG): container finished" podID="21a2f9bb-259e-4cd8-a738-36e9943aad75" containerID="e18b60bfe56df15698fa7f6182a479acb4a3dea7ad92e4d4fb3ac00e0b66bfb6" exitCode=0 Jul 22 17:13:47.174692 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:47.174511 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" Jul 22 17:13:47.174692 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:47.174502 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" event={"ID":"21a2f9bb-259e-4cd8-a738-36e9943aad75","Type":"ContainerDied","Data":"e18b60bfe56df15698fa7f6182a479acb4a3dea7ad92e4d4fb3ac00e0b66bfb6"} Jul 22 17:13:47.174692 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:47.174645 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z" event={"ID":"21a2f9bb-259e-4cd8-a738-36e9943aad75","Type":"ContainerDied","Data":"b03e1bf971d1b683d759d7de8fec66b52557a8420d8648c6539b7715fc563480"} Jul 22 17:13:47.174692 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:47.174677 2557 scope.go:117] "RemoveContainer" containerID="e18b60bfe56df15698fa7f6182a479acb4a3dea7ad92e4d4fb3ac00e0b66bfb6" Jul 22 17:13:47.186273 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:47.186251 2557 scope.go:117] "RemoveContainer" containerID="e18b60bfe56df15698fa7f6182a479acb4a3dea7ad92e4d4fb3ac00e0b66bfb6" Jul 22 17:13:47.186935 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:13:47.186911 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e18b60bfe56df15698fa7f6182a479acb4a3dea7ad92e4d4fb3ac00e0b66bfb6\": container with ID starting with e18b60bfe56df15698fa7f6182a479acb4a3dea7ad92e4d4fb3ac00e0b66bfb6 not found: ID does not exist" containerID="e18b60bfe56df15698fa7f6182a479acb4a3dea7ad92e4d4fb3ac00e0b66bfb6" Jul 22 17:13:47.186976 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:47.186949 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e18b60bfe56df15698fa7f6182a479acb4a3dea7ad92e4d4fb3ac00e0b66bfb6"} err="failed to get container status \"e18b60bfe56df15698fa7f6182a479acb4a3dea7ad92e4d4fb3ac00e0b66bfb6\": rpc error: code = NotFound desc = could not find container \"e18b60bfe56df15698fa7f6182a479acb4a3dea7ad92e4d4fb3ac00e0b66bfb6\": container with ID starting with e18b60bfe56df15698fa7f6182a479acb4a3dea7ad92e4d4fb3ac00e0b66bfb6 not found: ID does not exist" Jul 22 17:13:47.203273 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:47.203239 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z"] Jul 22 17:13:47.208265 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:47.208235 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-51445c0e-6bbcf567b-smj2z"] Jul 22 17:13:47.575841 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:47.575802 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21a2f9bb-259e-4cd8-a738-36e9943aad75" path="/var/lib/kubelet/pods/21a2f9bb-259e-4cd8-a738-36e9943aad75/volumes" Jul 22 17:13:49.718812 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:49.718782 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:51.144319 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:51.144282 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:51.166567 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:51.166534 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw"] Jul 22 17:13:51.172291 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:51.172266 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-51445c0e-d96qw"] Jul 22 17:13:51.577568 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:51.577535 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:51.580162 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:51.580130 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79653469-2dc3-4061-844b-a69a1481c2ee" path="/var/lib/kubelet/pods/79653469-2dc3-4061-844b-a69a1481c2ee/volumes" Jul 22 17:13:59.004952 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:59.004918 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:59.119239 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:59.119206 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:59.373503 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:59.373422 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:59.572588 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:59.572550 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:13:59.757720 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:13:59.757683 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:00.010120 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:00.010013 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:00.170656 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:00.170625 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:06.412248 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:06.412216 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:06.531305 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:06.531268 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:06.633474 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:06.633445 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:06.848211 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:06.848175 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:12.575885 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:12.575853 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:12.636669 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:12.636639 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:19.582234 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:19.582195 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:19.899658 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:19.899565 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:20.222982 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:20.222876 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:22.943661 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:22.943613 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:23.063452 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.063410 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:23.268776 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.268746 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9"] Jul 22 17:14:23.269154 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.269141 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a2f9bb-259e-4cd8-a738-36e9943aad75" containerName="maas-api" Jul 22 17:14:23.269206 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.269155 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a2f9bb-259e-4cd8-a738-36e9943aad75" containerName="maas-api" Jul 22 17:14:23.269206 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.269170 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79653469-2dc3-4061-844b-a69a1481c2ee" containerName="revoke-keys" Jul 22 17:14:23.269206 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.269177 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="79653469-2dc3-4061-844b-a69a1481c2ee" containerName="revoke-keys" Jul 22 17:14:23.269303 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.269256 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a2f9bb-259e-4cd8-a738-36e9943aad75" containerName="maas-api" Jul 22 17:14:23.269303 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.269264 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="79653469-2dc3-4061-844b-a69a1481c2ee" containerName="revoke-keys" Jul 22 17:14:23.272511 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.272494 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" Jul 22 17:14:23.276280 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.276262 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8sp7j\"" Jul 22 17:14:23.276393 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.276332 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c1b75e3f-serving-cert\"" Jul 22 17:14:23.293213 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.293188 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9"] Jul 22 17:14:23.324804 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.324761 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:23.417382 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.417352 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74754843-ca4c-4f00-b371-40df61074028-maas-api-tls\") pod \"maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9\" (UID: \"74754843-ca4c-4f00-b371-40df61074028\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" Jul 22 17:14:23.417559 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.417474 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvv9l\" (UniqueName: \"kubernetes.io/projected/74754843-ca4c-4f00-b371-40df61074028-kube-api-access-pvv9l\") pod \"maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9\" (UID: \"74754843-ca4c-4f00-b371-40df61074028\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" Jul 22 17:14:23.417559 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.417514 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74754843-ca4c-4f00-b371-40df61074028-openshift-service-ca\") pod \"maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9\" (UID: \"74754843-ca4c-4f00-b371-40df61074028\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" Jul 22 17:14:23.518005 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.517974 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74754843-ca4c-4f00-b371-40df61074028-openshift-service-ca\") pod \"maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9\" (UID: \"74754843-ca4c-4f00-b371-40df61074028\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" Jul 22 17:14:23.518205 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.518020 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74754843-ca4c-4f00-b371-40df61074028-maas-api-tls\") pod \"maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9\" (UID: \"74754843-ca4c-4f00-b371-40df61074028\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" Jul 22 17:14:23.518205 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.518132 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvv9l\" (UniqueName: \"kubernetes.io/projected/74754843-ca4c-4f00-b371-40df61074028-kube-api-access-pvv9l\") pod \"maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9\" (UID: \"74754843-ca4c-4f00-b371-40df61074028\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" Jul 22 17:14:23.518624 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.518592 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74754843-ca4c-4f00-b371-40df61074028-openshift-service-ca\") pod \"maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9\" (UID: \"74754843-ca4c-4f00-b371-40df61074028\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" Jul 22 17:14:23.520487 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.520441 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74754843-ca4c-4f00-b371-40df61074028-maas-api-tls\") pod \"maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9\" (UID: \"74754843-ca4c-4f00-b371-40df61074028\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" Jul 22 17:14:23.529350 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.529324 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvv9l\" (UniqueName: \"kubernetes.io/projected/74754843-ca4c-4f00-b371-40df61074028-kube-api-access-pvv9l\") pod \"maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9\" (UID: \"74754843-ca4c-4f00-b371-40df61074028\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" Jul 22 17:14:23.554839 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.554810 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:23.582418 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.582393 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" Jul 22 17:14:23.719551 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.719523 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9"] Jul 22 17:14:23.721939 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:14:23.721911 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74754843_ca4c_4f00_b371_40df61074028.slice/crio-82f405b464849c77ea50c01883a3abadd00714ea667a036e9f0c69251db270f1 WatchSource:0}: Error finding container 82f405b464849c77ea50c01883a3abadd00714ea667a036e9f0c69251db270f1: Status 404 returned error can't find the container with id 82f405b464849c77ea50c01883a3abadd00714ea667a036e9f0c69251db270f1 Jul 22 17:14:23.726793 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.726771 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 17:14:23.956589 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:23.956499 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:24.061225 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:24.061189 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:24.360692 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:24.360654 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" event={"ID":"74754843-ca4c-4f00-b371-40df61074028","Type":"ContainerStarted","Data":"82f405b464849c77ea50c01883a3abadd00714ea667a036e9f0c69251db270f1"} Jul 22 17:14:25.366758 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:25.366720 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" event={"ID":"74754843-ca4c-4f00-b371-40df61074028","Type":"ContainerStarted","Data":"7615f9abf9475adc7e94ce1306cce8ec9a0b1d1d8a9e681cc4014e6aaee77361"} Jul 22 17:14:25.367176 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:25.366859 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" Jul 22 17:14:25.410353 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:25.410302 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" podStartSLOduration=1.71057272 podStartE2EDuration="2.410285194s" totalImagesPullingTime="699.712474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:14:25.407676137 +0000 UTC m=+2030.384404454" watchObservedRunningTime="2026-07-22 17:14:25.410285194 +0000 UTC m=+2030.387013509" Jul 22 17:14:27.465934 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:27.465904 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:31.378308 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:31.378280 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" Jul 22 17:14:46.859752 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:46.859716 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:14:56.626134 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:14:56.626095 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:15:00.132913 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:00.132832 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745675-pz6pc"] Jul 22 17:15:00.137062 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:00.137038 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c1b75e3f-29745675-ll2fs"] Jul 22 17:15:00.137203 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:00.137139 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745675-pz6pc" Jul 22 17:15:00.139700 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:00.139680 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vln7b\"" Jul 22 17:15:00.140589 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:00.140525 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c1b75e3f-29745675-ll2fs" Jul 22 17:15:00.143728 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:00.143706 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745675-pz6pc"] Jul 22 17:15:00.146286 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:00.145854 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c1b75e3f-29745675-ll2fs"] Jul 22 17:15:00.151162 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:00.151146 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745675-pz6pc" Jul 22 17:15:00.158604 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:00.158588 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c1b75e3f-29745675-ll2fs" Jul 22 17:15:00.292627 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:00.292598 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745675-pz6pc"] Jul 22 17:15:00.295062 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:15:00.295017 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b085e8d_3a18_4ffe_ac43_1f21296d308f.slice/crio-080732e281d91b0ba0d30c6a70b949a1224e7a964b5c0b1aced302cc6c34f133 WatchSource:0}: Error finding container 080732e281d91b0ba0d30c6a70b949a1224e7a964b5c0b1aced302cc6c34f133: Status 404 returned error can't find the container with id 080732e281d91b0ba0d30c6a70b949a1224e7a964b5c0b1aced302cc6c34f133 Jul 22 17:15:00.323786 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:00.323750 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c1b75e3f-29745675-ll2fs"] Jul 22 17:15:00.326181 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:15:00.326158 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d2a53e_7741_49cf_a6b2_e15200a168a3.slice/crio-838f0108ab1cb1f0793e7a15133e0e81e2c38b1e137cdd6cece5c548683065e0 WatchSource:0}: Error finding container 838f0108ab1cb1f0793e7a15133e0e81e2c38b1e137cdd6cece5c548683065e0: Status 404 returned error can't find the container with id 838f0108ab1cb1f0793e7a15133e0e81e2c38b1e137cdd6cece5c548683065e0 Jul 22 17:15:00.548818 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:00.548781 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745675-pz6pc" event={"ID":"6b085e8d-3a18-4ffe-ac43-1f21296d308f","Type":"ContainerStarted","Data":"080732e281d91b0ba0d30c6a70b949a1224e7a964b5c0b1aced302cc6c34f133"} Jul 22 17:15:00.550117 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:00.550088 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c1b75e3f-29745675-ll2fs" event={"ID":"36d2a53e-7741-49cf-a6b2-e15200a168a3","Type":"ContainerStarted","Data":"838f0108ab1cb1f0793e7a15133e0e81e2c38b1e137cdd6cece5c548683065e0"} Jul 22 17:15:01.557321 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:01.557289 2557 generic.go:356] "Generic (PLEG): container finished" podID="6b085e8d-3a18-4ffe-ac43-1f21296d308f" containerID="8e54eef54ab35d7e3719999c5b13c4ae8c595b670ce2fb79575c8404add3370e" exitCode=0 Jul 22 17:15:01.557768 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:01.557377 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745675-pz6pc" event={"ID":"6b085e8d-3a18-4ffe-ac43-1f21296d308f","Type":"ContainerDied","Data":"8e54eef54ab35d7e3719999c5b13c4ae8c595b670ce2fb79575c8404add3370e"} Jul 22 17:15:01.558851 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:01.558826 2557 generic.go:356] "Generic (PLEG): container finished" podID="36d2a53e-7741-49cf-a6b2-e15200a168a3" containerID="4ba4167277fa17c57e19d47c4b2eefb2e562630190ad0ccd2ce3885eb7ef3f2d" exitCode=0 Jul 22 17:15:01.558959 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:01.558860 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c1b75e3f-29745675-ll2fs" event={"ID":"36d2a53e-7741-49cf-a6b2-e15200a168a3","Type":"ContainerDied","Data":"4ba4167277fa17c57e19d47c4b2eefb2e562630190ad0ccd2ce3885eb7ef3f2d"} Jul 22 17:15:02.730201 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:02.730171 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c1b75e3f-29745675-ll2fs" Jul 22 17:15:02.733351 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:02.733328 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745675-pz6pc" Jul 22 17:15:03.570556 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:03.570525 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745675-pz6pc" Jul 22 17:15:03.571882 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:03.571861 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c1b75e3f-29745675-ll2fs" Jul 22 17:15:03.576630 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:03.576601 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745675-pz6pc" event={"ID":"6b085e8d-3a18-4ffe-ac43-1f21296d308f","Type":"ContainerDied","Data":"080732e281d91b0ba0d30c6a70b949a1224e7a964b5c0b1aced302cc6c34f133"} Jul 22 17:15:03.576770 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:03.576651 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="080732e281d91b0ba0d30c6a70b949a1224e7a964b5c0b1aced302cc6c34f133" Jul 22 17:15:03.576770 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:03.576670 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c1b75e3f-29745675-ll2fs" event={"ID":"36d2a53e-7741-49cf-a6b2-e15200a168a3","Type":"ContainerDied","Data":"838f0108ab1cb1f0793e7a15133e0e81e2c38b1e137cdd6cece5c548683065e0"} Jul 22 17:15:03.576770 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:03.576696 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="838f0108ab1cb1f0793e7a15133e0e81e2c38b1e137cdd6cece5c548683065e0" Jul 22 17:15:21.609469 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:21.609428 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2"] Jul 22 17:15:21.610005 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:21.609985 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36d2a53e-7741-49cf-a6b2-e15200a168a3" containerName="cleanup" Jul 22 17:15:21.610128 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:21.610008 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="36d2a53e-7741-49cf-a6b2-e15200a168a3" containerName="cleanup" Jul 22 17:15:21.610128 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:21.610075 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b085e8d-3a18-4ffe-ac43-1f21296d308f" containerName="cleanup" Jul 22 17:15:21.610128 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:21.610087 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b085e8d-3a18-4ffe-ac43-1f21296d308f" containerName="cleanup" Jul 22 17:15:21.610278 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:21.610191 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="36d2a53e-7741-49cf-a6b2-e15200a168a3" containerName="cleanup" Jul 22 17:15:21.610278 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:21.610209 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b085e8d-3a18-4ffe-ac43-1f21296d308f" containerName="cleanup" Jul 22 17:15:21.613317 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:21.613294 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2" Jul 22 17:15:21.616267 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:21.616241 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vln7b\"" Jul 22 17:15:21.621292 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:21.621267 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2"] Jul 22 17:15:21.657185 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:21.657154 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15196e87-d0f0-4105-8e4d-7151dbe509ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2\" (UID: \"15196e87-d0f0-4105-8e4d-7151dbe509ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2" Jul 22 17:15:21.757967 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:21.757934 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15196e87-d0f0-4105-8e4d-7151dbe509ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2\" (UID: \"15196e87-d0f0-4105-8e4d-7151dbe509ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2" Jul 22 17:15:21.758570 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:21.758549 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15196e87-d0f0-4105-8e4d-7151dbe509ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2\" (UID: \"15196e87-d0f0-4105-8e4d-7151dbe509ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2" Jul 22 17:15:21.926615 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:21.926530 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2" Jul 22 17:15:22.060002 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:22.059977 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2"] Jul 22 17:15:22.062746 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:15:22.062716 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15196e87_d0f0_4105_8e4d_7151dbe509ae.slice/crio-d8b10ac440a939cdb571c70407ac811d98ed96941616afa256d5723e8acddf38 WatchSource:0}: Error finding container d8b10ac440a939cdb571c70407ac811d98ed96941616afa256d5723e8acddf38: Status 404 returned error can't find the container with id d8b10ac440a939cdb571c70407ac811d98ed96941616afa256d5723e8acddf38 Jul 22 17:15:22.671459 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:22.671361 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2" event={"ID":"15196e87-d0f0-4105-8e4d-7151dbe509ae","Type":"ContainerStarted","Data":"5cba3aedecf348c945d8a5e3f5ae34f8427eea6e23067a752a3d083b4b79c830"} Jul 22 17:15:22.671459 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:22.671433 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2" event={"ID":"15196e87-d0f0-4105-8e4d-7151dbe509ae","Type":"ContainerStarted","Data":"d8b10ac440a939cdb571c70407ac811d98ed96941616afa256d5723e8acddf38"} Jul 22 17:15:22.689767 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:22.689720 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2" podStartSLOduration=1.689705865 podStartE2EDuration="1.689705865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:15:22.688091851 +0000 UTC m=+2087.664820167" watchObservedRunningTime="2026-07-22 17:15:22.689705865 +0000 UTC m=+2087.666434181" Jul 22 17:15:23.678206 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:23.678163 2557 generic.go:356] "Generic (PLEG): container finished" podID="15196e87-d0f0-4105-8e4d-7151dbe509ae" containerID="5cba3aedecf348c945d8a5e3f5ae34f8427eea6e23067a752a3d083b4b79c830" exitCode=0 Jul 22 17:15:23.678577 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:23.678200 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2" event={"ID":"15196e87-d0f0-4105-8e4d-7151dbe509ae","Type":"ContainerDied","Data":"5cba3aedecf348c945d8a5e3f5ae34f8427eea6e23067a752a3d083b4b79c830"} Jul 22 17:15:24.816137 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:24.816109 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2" Jul 22 17:15:24.888366 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:24.888336 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15196e87-d0f0-4105-8e4d-7151dbe509ae-maas-api-service-ca\") pod \"15196e87-d0f0-4105-8e4d-7151dbe509ae\" (UID: \"15196e87-d0f0-4105-8e4d-7151dbe509ae\") " Jul 22 17:15:24.888701 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:24.888679 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15196e87-d0f0-4105-8e4d-7151dbe509ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "15196e87-d0f0-4105-8e4d-7151dbe509ae" (UID: "15196e87-d0f0-4105-8e4d-7151dbe509ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:15:24.989798 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:24.989768 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15196e87-d0f0-4105-8e4d-7151dbe509ae-maas-api-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:15:25.689524 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:25.689492 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2" event={"ID":"15196e87-d0f0-4105-8e4d-7151dbe509ae","Type":"ContainerDied","Data":"d8b10ac440a939cdb571c70407ac811d98ed96941616afa256d5723e8acddf38"} Jul 22 17:15:25.689524 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:25.689511 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2" Jul 22 17:15:25.689524 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:25.689540 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8b10ac440a939cdb571c70407ac811d98ed96941616afa256d5723e8acddf38" Jul 22 17:15:35.696254 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:35.696225 2557 scope.go:117] "RemoveContainer" containerID="7025f63a9e4169b7716a826e3ee571394d43605aa30d1d8dcbbdcfbdbd833999" Jul 22 17:15:37.042237 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.042202 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9"] Jul 22 17:15:37.042709 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.042482 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" podUID="74754843-ca4c-4f00-b371-40df61074028" containerName="maas-api" containerID="cri-o://7615f9abf9475adc7e94ce1306cce8ec9a0b1d1d8a9e681cc4014e6aaee77361" gracePeriod=30 Jul 22 17:15:37.102898 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.102858 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c1b75e3f-29745675-ll2fs"] Jul 22 17:15:37.107904 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.107872 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c1b75e3f-29745675-ll2fs"] Jul 22 17:15:37.135784 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.135749 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:15:37.290242 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.290214 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" Jul 22 17:15:37.409725 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.409621 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74754843-ca4c-4f00-b371-40df61074028-openshift-service-ca\") pod \"74754843-ca4c-4f00-b371-40df61074028\" (UID: \"74754843-ca4c-4f00-b371-40df61074028\") " Jul 22 17:15:37.409725 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.409686 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74754843-ca4c-4f00-b371-40df61074028-maas-api-tls\") pod \"74754843-ca4c-4f00-b371-40df61074028\" (UID: \"74754843-ca4c-4f00-b371-40df61074028\") " Jul 22 17:15:37.409969 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.409740 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvv9l\" (UniqueName: \"kubernetes.io/projected/74754843-ca4c-4f00-b371-40df61074028-kube-api-access-pvv9l\") pod \"74754843-ca4c-4f00-b371-40df61074028\" (UID: \"74754843-ca4c-4f00-b371-40df61074028\") " Jul 22 17:15:37.410108 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.410085 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74754843-ca4c-4f00-b371-40df61074028-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "74754843-ca4c-4f00-b371-40df61074028" (UID: "74754843-ca4c-4f00-b371-40df61074028"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:15:37.411915 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.411891 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74754843-ca4c-4f00-b371-40df61074028-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "74754843-ca4c-4f00-b371-40df61074028" (UID: "74754843-ca4c-4f00-b371-40df61074028"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:15:37.411915 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.411906 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74754843-ca4c-4f00-b371-40df61074028-kube-api-access-pvv9l" (OuterVolumeSpecName: "kube-api-access-pvv9l") pod "74754843-ca4c-4f00-b371-40df61074028" (UID: "74754843-ca4c-4f00-b371-40df61074028"). InnerVolumeSpecName "kube-api-access-pvv9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:15:37.511221 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.511191 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvv9l\" (UniqueName: \"kubernetes.io/projected/74754843-ca4c-4f00-b371-40df61074028-kube-api-access-pvv9l\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:15:37.511221 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.511218 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74754843-ca4c-4f00-b371-40df61074028-openshift-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:15:37.511221 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.511228 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74754843-ca4c-4f00-b371-40df61074028-maas-api-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:15:37.575154 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.575123 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36d2a53e-7741-49cf-a6b2-e15200a168a3" path="/var/lib/kubelet/pods/36d2a53e-7741-49cf-a6b2-e15200a168a3/volumes" Jul 22 17:15:37.755199 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.755143 2557 generic.go:356] "Generic (PLEG): container finished" podID="74754843-ca4c-4f00-b371-40df61074028" containerID="7615f9abf9475adc7e94ce1306cce8ec9a0b1d1d8a9e681cc4014e6aaee77361" exitCode=0 Jul 22 17:15:37.755199 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.755208 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" Jul 22 17:15:37.755439 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.755229 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" event={"ID":"74754843-ca4c-4f00-b371-40df61074028","Type":"ContainerDied","Data":"7615f9abf9475adc7e94ce1306cce8ec9a0b1d1d8a9e681cc4014e6aaee77361"} Jul 22 17:15:37.755439 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.755307 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9" event={"ID":"74754843-ca4c-4f00-b371-40df61074028","Type":"ContainerDied","Data":"82f405b464849c77ea50c01883a3abadd00714ea667a036e9f0c69251db270f1"} Jul 22 17:15:37.755439 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.755333 2557 scope.go:117] "RemoveContainer" containerID="7615f9abf9475adc7e94ce1306cce8ec9a0b1d1d8a9e681cc4014e6aaee77361" Jul 22 17:15:37.764441 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.764422 2557 scope.go:117] "RemoveContainer" containerID="7615f9abf9475adc7e94ce1306cce8ec9a0b1d1d8a9e681cc4014e6aaee77361" Jul 22 17:15:37.764711 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:15:37.764693 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7615f9abf9475adc7e94ce1306cce8ec9a0b1d1d8a9e681cc4014e6aaee77361\": container with ID starting with 7615f9abf9475adc7e94ce1306cce8ec9a0b1d1d8a9e681cc4014e6aaee77361 not found: ID does not exist" containerID="7615f9abf9475adc7e94ce1306cce8ec9a0b1d1d8a9e681cc4014e6aaee77361" Jul 22 17:15:37.764795 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.764715 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7615f9abf9475adc7e94ce1306cce8ec9a0b1d1d8a9e681cc4014e6aaee77361"} err="failed to get container status \"7615f9abf9475adc7e94ce1306cce8ec9a0b1d1d8a9e681cc4014e6aaee77361\": rpc error: code = NotFound desc = could not find container \"7615f9abf9475adc7e94ce1306cce8ec9a0b1d1d8a9e681cc4014e6aaee77361\": container with ID starting with 7615f9abf9475adc7e94ce1306cce8ec9a0b1d1d8a9e681cc4014e6aaee77361 not found: ID does not exist" Jul 22 17:15:37.774582 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.774558 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9"] Jul 22 17:15:37.777639 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:37.777618 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1b75e3f-797bd48f69-mzqh9"] Jul 22 17:15:39.575274 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:39.575238 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74754843-ca4c-4f00-b371-40df61074028" path="/var/lib/kubelet/pods/74754843-ca4c-4f00-b371-40df61074028/volumes" Jul 22 17:15:41.682704 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:41.682674 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2"] Jul 22 17:15:41.687015 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:41.686988 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1b75e3f-rr4j2"] Jul 22 17:15:41.690461 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:41.690427 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:15:43.575057 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:43.575007 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15196e87-d0f0-4105-8e4d-7151dbe509ae" path="/var/lib/kubelet/pods/15196e87-d0f0-4105-8e4d-7151dbe509ae/volumes" Jul 22 17:15:44.010179 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:44.010150 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:15:48.997913 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:15:48.997858 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:10.220594 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:10.220562 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:10.318496 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:10.318463 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:25.537676 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:25.537644 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:26.166238 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:26.166208 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:31.645807 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:31.645770 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:31.746368 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:31.746335 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:31.958436 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:31.958355 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:32.172588 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:32.172553 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:33.256452 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:33.256411 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:33.350504 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:33.350470 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:33.450846 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:33.450781 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:34.155388 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:34.155347 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:34.254743 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:34.254714 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:34.355489 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:34.355450 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:35.716600 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:35.716570 2557 scope.go:117] "RemoveContainer" containerID="5a88607ad4555fbea3f187a2e6f3ba290ef7bf58e03a2ac3fac0de2980396fcf" Jul 22 17:16:35.749238 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:35.749211 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:41.459348 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:41.459313 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:41.560011 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:41.559973 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:42.172704 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:42.172661 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:48.984012 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:48.983976 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:49.136399 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:49.136366 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:49.234450 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:49.234359 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:16:59.711652 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:16:59.711615 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:00.024175 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:00.024143 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:21.033548 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:21.033514 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:27.709117 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:27.709085 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:27.813222 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:27.813187 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:28.135296 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:28.135251 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:28.493081 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:28.493049 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:28.865458 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:28.865358 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:31.800141 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:31.800101 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:31.894670 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:31.894635 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:31.928576 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:31.928542 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558"] Jul 22 17:17:31.928944 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:31.928931 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74754843-ca4c-4f00-b371-40df61074028" containerName="maas-api" Jul 22 17:17:31.928988 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:31.928945 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="74754843-ca4c-4f00-b371-40df61074028" containerName="maas-api" Jul 22 17:17:31.928988 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:31.928967 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15196e87-d0f0-4105-8e4d-7151dbe509ae" containerName="revoke-keys" Jul 22 17:17:31.928988 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:31.928972 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="15196e87-d0f0-4105-8e4d-7151dbe509ae" containerName="revoke-keys" Jul 22 17:17:31.929113 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:31.929042 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="74754843-ca4c-4f00-b371-40df61074028" containerName="maas-api" Jul 22 17:17:31.929113 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:31.929051 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="15196e87-d0f0-4105-8e4d-7151dbe509ae" containerName="revoke-keys" Jul 22 17:17:31.932317 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:31.932301 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:17:31.935523 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:31.935497 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8sp7j\"" Jul 22 17:17:31.935523 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:31.935497 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-668a76-serving-cert\"" Jul 22 17:17:31.943898 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:31.943874 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558"] Jul 22 17:17:32.038107 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.038071 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:32.084626 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.084543 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/363f48c4-bbdb-463c-bdc6-8bdec90566c4-maas-api-tls\") pod \"maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558\" (UID: \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:17:32.084626 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.084587 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/363f48c4-bbdb-463c-bdc6-8bdec90566c4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558\" (UID: \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:17:32.084859 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.084718 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-584kz\" (UniqueName: \"kubernetes.io/projected/363f48c4-bbdb-463c-bdc6-8bdec90566c4-kube-api-access-584kz\") pod \"maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558\" (UID: \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:17:32.185548 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.185511 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-584kz\" (UniqueName: \"kubernetes.io/projected/363f48c4-bbdb-463c-bdc6-8bdec90566c4-kube-api-access-584kz\") pod \"maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558\" (UID: \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:17:32.185732 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.185571 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/363f48c4-bbdb-463c-bdc6-8bdec90566c4-maas-api-tls\") pod \"maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558\" (UID: \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:17:32.185732 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.185597 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/363f48c4-bbdb-463c-bdc6-8bdec90566c4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558\" (UID: \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:17:32.185815 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:17:32.185723 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-serving-cert: secret "maas-api-e2e-inf-a-668a76-serving-cert" not found Jul 22 17:17:32.185815 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:17:32.185797 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/363f48c4-bbdb-463c-bdc6-8bdec90566c4-maas-api-tls podName:363f48c4-bbdb-463c-bdc6-8bdec90566c4 nodeName:}" failed. No retries permitted until 2026-07-22 17:17:32.685776287 +0000 UTC m=+2217.662504586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/363f48c4-bbdb-463c-bdc6-8bdec90566c4-maas-api-tls") pod "maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" (UID: "363f48c4-bbdb-463c-bdc6-8bdec90566c4") : secret "maas-api-e2e-inf-a-668a76-serving-cert" not found Jul 22 17:17:32.186232 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.186216 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/363f48c4-bbdb-463c-bdc6-8bdec90566c4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558\" (UID: \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:17:32.196219 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.196194 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-584kz\" (UniqueName: \"kubernetes.io/projected/363f48c4-bbdb-463c-bdc6-8bdec90566c4-kube-api-access-584kz\") pod \"maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558\" (UID: \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:17:32.248347 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.248314 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:32.634519 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.634490 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:32.690417 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.690378 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/363f48c4-bbdb-463c-bdc6-8bdec90566c4-maas-api-tls\") pod \"maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558\" (UID: \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:17:32.692849 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.692822 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/363f48c4-bbdb-463c-bdc6-8bdec90566c4-maas-api-tls\") pod \"maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558\" (UID: \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:17:32.745171 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.745139 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:32.842498 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.842469 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:17:32.984514 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:32.984485 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558"] Jul 22 17:17:32.986175 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:17:32.986151 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363f48c4_bbdb_463c_bdc6_8bdec90566c4.slice/crio-286766607b2145e307cc83496b8b319bc2ea0b196602f6858f02cc1934236dce WatchSource:0}: Error finding container 286766607b2145e307cc83496b8b319bc2ea0b196602f6858f02cc1934236dce: Status 404 returned error can't find the container with id 286766607b2145e307cc83496b8b319bc2ea0b196602f6858f02cc1934236dce Jul 22 17:17:33.353769 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:33.353729 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" event={"ID":"363f48c4-bbdb-463c-bdc6-8bdec90566c4","Type":"ContainerStarted","Data":"286766607b2145e307cc83496b8b319bc2ea0b196602f6858f02cc1934236dce"} Jul 22 17:17:34.360737 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:34.360695 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" event={"ID":"363f48c4-bbdb-463c-bdc6-8bdec90566c4","Type":"ContainerStarted","Data":"cf1311ec80a12e3e6d331de8be6a240bd22eaac33e0dee97444f2299612ad986"} Jul 22 17:17:34.361197 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:34.360756 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:17:34.377833 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:34.377783 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" podStartSLOduration=2.7925429299999998 podStartE2EDuration="3.377763491s" totalImagesPullingTime="585.220561ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:17:34.377546759 +0000 UTC m=+2219.354275088" watchObservedRunningTime="2026-07-22 17:17:34.377763491 +0000 UTC m=+2219.354491809" Jul 22 17:17:35.741012 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:35.740979 2557 scope.go:117] "RemoveContainer" containerID="2c80e21525db1923870d383262f874db0bdc72627c233669a7faf61ce2e09592" Jul 22 17:17:35.753448 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:35.753414 2557 scope.go:117] "RemoveContainer" containerID="c67f03a3ca27a7bff786de6a0846ef6e52c890559b07f5a13428fb082f737d26" Jul 22 17:17:36.345592 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:36.345550 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:36.681078 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:36.680955 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:36.960489 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:36.960400 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:37.206202 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:37.206168 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:39.793902 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:39.793862 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:39.878175 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:39.878140 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk"] Jul 22 17:17:39.883759 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:39.883732 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:17:39.888194 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:39.888170 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f854fd-serving-cert\"" Jul 22 17:17:39.896813 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:39.896785 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk"] Jul 22 17:17:39.897679 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:39.897651 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:39.996075 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:39.996014 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:40.063094 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.062974 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfdnx\" (UniqueName: \"kubernetes.io/projected/9278e3b6-81cb-4416-b7ff-5ac458ea173f-kube-api-access-nfdnx\") pod \"maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk\" (UID: \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:17:40.063260 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.063103 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9278e3b6-81cb-4416-b7ff-5ac458ea173f-maas-api-tls\") pod \"maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk\" (UID: \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:17:40.063260 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.063146 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9278e3b6-81cb-4416-b7ff-5ac458ea173f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk\" (UID: \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:17:40.164245 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.164207 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9278e3b6-81cb-4416-b7ff-5ac458ea173f-maas-api-tls\") pod \"maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk\" (UID: \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:17:40.164245 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.164246 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9278e3b6-81cb-4416-b7ff-5ac458ea173f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk\" (UID: \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:17:40.164455 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.164309 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfdnx\" (UniqueName: \"kubernetes.io/projected/9278e3b6-81cb-4416-b7ff-5ac458ea173f-kube-api-access-nfdnx\") pod \"maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk\" (UID: \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:17:40.164455 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:17:40.164368 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-serving-cert: secret "maas-api-e2e-inf-b-f854fd-serving-cert" not found Jul 22 17:17:40.164455 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:17:40.164446 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9278e3b6-81cb-4416-b7ff-5ac458ea173f-maas-api-tls podName:9278e3b6-81cb-4416-b7ff-5ac458ea173f nodeName:}" failed. No retries permitted until 2026-07-22 17:17:40.664425436 +0000 UTC m=+2225.641153744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9278e3b6-81cb-4416-b7ff-5ac458ea173f-maas-api-tls") pod "maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" (UID: "9278e3b6-81cb-4416-b7ff-5ac458ea173f") : secret "maas-api-e2e-inf-b-f854fd-serving-cert" not found Jul 22 17:17:40.164968 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.164946 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9278e3b6-81cb-4416-b7ff-5ac458ea173f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk\" (UID: \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:17:40.176655 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.176630 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfdnx\" (UniqueName: \"kubernetes.io/projected/9278e3b6-81cb-4416-b7ff-5ac458ea173f-kube-api-access-nfdnx\") pod \"maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk\" (UID: \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:17:40.271219 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.271189 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:40.371315 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.371245 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:17:40.669633 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.669549 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9278e3b6-81cb-4416-b7ff-5ac458ea173f-maas-api-tls\") pod \"maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk\" (UID: \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:17:40.672014 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.671988 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9278e3b6-81cb-4416-b7ff-5ac458ea173f-maas-api-tls\") pod \"maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk\" (UID: \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:17:40.746077 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.746019 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:40.796106 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.796074 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:17:40.846665 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.846633 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:40.933549 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:40.933524 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk"] Jul 22 17:17:40.935782 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:17:40.935753 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9278e3b6_81cb_4416_b7ff_5ac458ea173f.slice/crio-a56396259fe1c9c278c120acf6b414ef7ed0ae5c4bc02cd5ac8f98477c1ac043 WatchSource:0}: Error finding container a56396259fe1c9c278c120acf6b414ef7ed0ae5c4bc02cd5ac8f98477c1ac043: Status 404 returned error can't find the container with id a56396259fe1c9c278c120acf6b414ef7ed0ae5c4bc02cd5ac8f98477c1ac043 Jul 22 17:17:41.397483 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:41.397446 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" event={"ID":"9278e3b6-81cb-4416-b7ff-5ac458ea173f","Type":"ContainerStarted","Data":"b2e9ded686fba968ef4e2f8b7f9177a23a848a685f955a9dc0d6f8e3b021f51d"} Jul 22 17:17:41.397649 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:41.397512 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" event={"ID":"9278e3b6-81cb-4416-b7ff-5ac458ea173f","Type":"ContainerStarted","Data":"a56396259fe1c9c278c120acf6b414ef7ed0ae5c4bc02cd5ac8f98477c1ac043"} Jul 22 17:17:41.397649 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:41.397619 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:17:41.417402 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:41.417356 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" podStartSLOduration=2.093821643 podStartE2EDuration="2.41732538s" totalImagesPullingTime="323.503737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:17:41.415898073 +0000 UTC m=+2226.392626401" watchObservedRunningTime="2026-07-22 17:17:41.41732538 +0000 UTC m=+2226.394053716" Jul 22 17:17:44.309903 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:44.309862 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:47.408560 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:47.408532 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:17:50.530746 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:50.530682 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:53.466108 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:53.466075 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:17:53.563871 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:17:53.563816 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:18:01.490062 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:01.489963 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:18:01.590571 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:01.590525 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:18:15.327190 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:15.327153 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:18:41.190283 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:41.190243 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:18:41.251427 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:41.251393 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:18:53.585631 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:53.585598 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:18:54.631667 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:54.631638 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:18:54.751169 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:54.751135 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg"] Jul 22 17:18:54.754951 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:54.754912 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg" Jul 22 17:18:54.758381 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:54.758359 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vln7b\"" Jul 22 17:18:54.765640 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:54.765612 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg"] Jul 22 17:18:54.882293 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:54.882195 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8429909b-191a-47b6-92bf-228764bd7803-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-668a76-s5brg\" (UID: \"8429909b-191a-47b6-92bf-228764bd7803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg" Jul 22 17:18:54.983176 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:54.983136 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8429909b-191a-47b6-92bf-228764bd7803-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-668a76-s5brg\" (UID: \"8429909b-191a-47b6-92bf-228764bd7803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg" Jul 22 17:18:54.983801 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:54.983779 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8429909b-191a-47b6-92bf-228764bd7803-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-668a76-s5brg\" (UID: \"8429909b-191a-47b6-92bf-228764bd7803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg" Jul 22 17:18:55.069748 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:55.069717 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg" Jul 22 17:18:55.204284 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:55.204247 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg"] Jul 22 17:18:55.206186 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:18:55.206152 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8429909b_191a_47b6_92bf_228764bd7803.slice/crio-1049c8cfb386d90199558b204c8599beaca761fa618aac5c2542fa6338742958 WatchSource:0}: Error finding container 1049c8cfb386d90199558b204c8599beaca761fa618aac5c2542fa6338742958: Status 404 returned error can't find the container with id 1049c8cfb386d90199558b204c8599beaca761fa618aac5c2542fa6338742958 Jul 22 17:18:55.386420 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:55.386362 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:18:55.488366 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:55.488326 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:18:55.790480 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:55.790442 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg" event={"ID":"8429909b-191a-47b6-92bf-228764bd7803","Type":"ContainerStarted","Data":"d9dfe5d9a75ea7f3fecabd935cd320b5c89302af40aebd445b0ee8467c69d408"} Jul 22 17:18:55.790870 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:55.790493 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg" event={"ID":"8429909b-191a-47b6-92bf-228764bd7803","Type":"ContainerStarted","Data":"1049c8cfb386d90199558b204c8599beaca761fa618aac5c2542fa6338742958"} Jul 22 17:18:55.810645 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:55.810592 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg" podStartSLOduration=1.810576199 podStartE2EDuration="1.810576199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:18:55.808605024 +0000 UTC m=+2300.785333340" watchObservedRunningTime="2026-07-22 17:18:55.810576199 +0000 UTC m=+2300.787304571" Jul 22 17:18:56.797219 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:56.797187 2557 generic.go:356] "Generic (PLEG): container finished" podID="8429909b-191a-47b6-92bf-228764bd7803" containerID="d9dfe5d9a75ea7f3fecabd935cd320b5c89302af40aebd445b0ee8467c69d408" exitCode=0 Jul 22 17:18:56.797649 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:56.797281 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg" event={"ID":"8429909b-191a-47b6-92bf-228764bd7803","Type":"ContainerDied","Data":"d9dfe5d9a75ea7f3fecabd935cd320b5c89302af40aebd445b0ee8467c69d408"} Jul 22 17:18:57.934403 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:57.934377 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg" Jul 22 17:18:58.110926 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:58.110846 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8429909b-191a-47b6-92bf-228764bd7803-maas-api-service-ca\") pod \"8429909b-191a-47b6-92bf-228764bd7803\" (UID: \"8429909b-191a-47b6-92bf-228764bd7803\") " Jul 22 17:18:58.111247 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:58.111226 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8429909b-191a-47b6-92bf-228764bd7803-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8429909b-191a-47b6-92bf-228764bd7803" (UID: "8429909b-191a-47b6-92bf-228764bd7803"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:18:58.218363 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:58.213176 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8429909b-191a-47b6-92bf-228764bd7803-maas-api-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:18:58.808448 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:58.808419 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg" Jul 22 17:18:58.808448 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:58.808437 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg" event={"ID":"8429909b-191a-47b6-92bf-228764bd7803","Type":"ContainerDied","Data":"1049c8cfb386d90199558b204c8599beaca761fa618aac5c2542fa6338742958"} Jul 22 17:18:58.808661 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:18:58.808485 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1049c8cfb386d90199558b204c8599beaca761fa618aac5c2542fa6338742958" Jul 22 17:19:05.297743 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.297715 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558"] Jul 22 17:19:05.298262 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.297919 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" podUID="363f48c4-bbdb-463c-bdc6-8bdec90566c4" containerName="maas-api" containerID="cri-o://cf1311ec80a12e3e6d331de8be6a240bd22eaac33e0dee97444f2299612ad986" gracePeriod=30 Jul 22 17:19:05.316792 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.316758 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:05.367236 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.367184 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" podUID="363f48c4-bbdb-463c-bdc6-8bdec90566c4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.53:8443/health\": dial tcp 10.134.0.53:8443: connect: connection refused" Jul 22 17:19:05.548210 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.548154 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:19:05.577704 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.577662 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/363f48c4-bbdb-463c-bdc6-8bdec90566c4-openshift-service-ca\") pod \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\" (UID: \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\") " Jul 22 17:19:05.577851 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.577739 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/363f48c4-bbdb-463c-bdc6-8bdec90566c4-maas-api-tls\") pod \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\" (UID: \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\") " Jul 22 17:19:05.577851 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.577786 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-584kz\" (UniqueName: \"kubernetes.io/projected/363f48c4-bbdb-463c-bdc6-8bdec90566c4-kube-api-access-584kz\") pod \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\" (UID: \"363f48c4-bbdb-463c-bdc6-8bdec90566c4\") " Jul 22 17:19:05.578989 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.578962 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363f48c4-bbdb-463c-bdc6-8bdec90566c4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "363f48c4-bbdb-463c-bdc6-8bdec90566c4" (UID: "363f48c4-bbdb-463c-bdc6-8bdec90566c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:19:05.584415 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.584178 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363f48c4-bbdb-463c-bdc6-8bdec90566c4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "363f48c4-bbdb-463c-bdc6-8bdec90566c4" (UID: "363f48c4-bbdb-463c-bdc6-8bdec90566c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:19:05.584545 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.584508 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363f48c4-bbdb-463c-bdc6-8bdec90566c4-kube-api-access-584kz" (OuterVolumeSpecName: "kube-api-access-584kz") pod "363f48c4-bbdb-463c-bdc6-8bdec90566c4" (UID: "363f48c4-bbdb-463c-bdc6-8bdec90566c4"). InnerVolumeSpecName "kube-api-access-584kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:19:05.679328 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.679294 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/363f48c4-bbdb-463c-bdc6-8bdec90566c4-openshift-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:19:05.679328 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.679323 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/363f48c4-bbdb-463c-bdc6-8bdec90566c4-maas-api-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:19:05.679328 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.679333 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-584kz\" (UniqueName: \"kubernetes.io/projected/363f48c4-bbdb-463c-bdc6-8bdec90566c4-kube-api-access-584kz\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:19:05.845391 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.845300 2557 generic.go:356] "Generic (PLEG): container finished" podID="363f48c4-bbdb-463c-bdc6-8bdec90566c4" containerID="cf1311ec80a12e3e6d331de8be6a240bd22eaac33e0dee97444f2299612ad986" exitCode=0 Jul 22 17:19:05.845391 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.845369 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" Jul 22 17:19:05.845391 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.845371 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" event={"ID":"363f48c4-bbdb-463c-bdc6-8bdec90566c4","Type":"ContainerDied","Data":"cf1311ec80a12e3e6d331de8be6a240bd22eaac33e0dee97444f2299612ad986"} Jul 22 17:19:05.845631 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.845427 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558" event={"ID":"363f48c4-bbdb-463c-bdc6-8bdec90566c4","Type":"ContainerDied","Data":"286766607b2145e307cc83496b8b319bc2ea0b196602f6858f02cc1934236dce"} Jul 22 17:19:05.845631 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.845448 2557 scope.go:117] "RemoveContainer" containerID="cf1311ec80a12e3e6d331de8be6a240bd22eaac33e0dee97444f2299612ad986" Jul 22 17:19:05.854313 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.854293 2557 scope.go:117] "RemoveContainer" containerID="cf1311ec80a12e3e6d331de8be6a240bd22eaac33e0dee97444f2299612ad986" Jul 22 17:19:05.854594 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:19:05.854574 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1311ec80a12e3e6d331de8be6a240bd22eaac33e0dee97444f2299612ad986\": container with ID starting with cf1311ec80a12e3e6d331de8be6a240bd22eaac33e0dee97444f2299612ad986 not found: ID does not exist" containerID="cf1311ec80a12e3e6d331de8be6a240bd22eaac33e0dee97444f2299612ad986" Jul 22 17:19:05.854669 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.854598 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1311ec80a12e3e6d331de8be6a240bd22eaac33e0dee97444f2299612ad986"} err="failed to get container status \"cf1311ec80a12e3e6d331de8be6a240bd22eaac33e0dee97444f2299612ad986\": rpc error: code = NotFound desc = could not find container \"cf1311ec80a12e3e6d331de8be6a240bd22eaac33e0dee97444f2299612ad986\": container with ID starting with cf1311ec80a12e3e6d331de8be6a240bd22eaac33e0dee97444f2299612ad986 not found: ID does not exist" Jul 22 17:19:05.874772 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.874742 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558"] Jul 22 17:19:05.879942 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:05.879917 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-668a76-67fc4bc98f-6s558"] Jul 22 17:19:07.575080 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:07.575042 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="363f48c4-bbdb-463c-bdc6-8bdec90566c4" path="/var/lib/kubelet/pods/363f48c4-bbdb-463c-bdc6-8bdec90566c4/volumes" Jul 22 17:19:09.851299 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:09.851257 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:09.892362 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:09.892330 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg"] Jul 22 17:19:09.903114 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:09.903087 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-668a76-s5brg"] Jul 22 17:19:10.311093 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:10.311050 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:11.574961 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:11.574914 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8429909b-191a-47b6-92bf-228764bd7803" path="/var/lib/kubelet/pods/8429909b-191a-47b6-92bf-228764bd7803/volumes" Jul 22 17:19:16.991303 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:16.991264 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:17.103098 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.103057 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:17.288809 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.288719 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:17.519571 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.519533 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:17.673982 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.673896 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg"] Jul 22 17:19:17.674359 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.674341 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8429909b-191a-47b6-92bf-228764bd7803" containerName="revoke-keys" Jul 22 17:19:17.674359 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.674360 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8429909b-191a-47b6-92bf-228764bd7803" containerName="revoke-keys" Jul 22 17:19:17.674468 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.674382 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="363f48c4-bbdb-463c-bdc6-8bdec90566c4" containerName="maas-api" Jul 22 17:19:17.674468 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.674389 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="363f48c4-bbdb-463c-bdc6-8bdec90566c4" containerName="maas-api" Jul 22 17:19:17.674535 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.674468 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="363f48c4-bbdb-463c-bdc6-8bdec90566c4" containerName="maas-api" Jul 22 17:19:17.674535 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.674484 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8429909b-191a-47b6-92bf-228764bd7803" containerName="revoke-keys" Jul 22 17:19:17.677890 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.677863 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg" Jul 22 17:19:17.681213 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.681185 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vln7b\"" Jul 22 17:19:17.690788 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.690763 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg"] Jul 22 17:19:17.791973 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.791935 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c546c147-b513-4d3f-ab87-4c0822e21d12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg\" (UID: \"c546c147-b513-4d3f-ab87-4c0822e21d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg" Jul 22 17:19:17.892884 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.892847 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c546c147-b513-4d3f-ab87-4c0822e21d12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg\" (UID: \"c546c147-b513-4d3f-ab87-4c0822e21d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg" Jul 22 17:19:17.893445 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.893427 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c546c147-b513-4d3f-ab87-4c0822e21d12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg\" (UID: \"c546c147-b513-4d3f-ab87-4c0822e21d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg" Jul 22 17:19:17.990081 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:17.990051 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg" Jul 22 17:19:18.127829 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:18.127800 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg"] Jul 22 17:19:18.130267 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:19:18.130238 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc546c147_b513_4d3f_ab87_4c0822e21d12.slice/crio-72e7fd13001c1ad370c8cba2c2aeb2efe27e402d1dfba1321e256ca30961ef0c WatchSource:0}: Error finding container 72e7fd13001c1ad370c8cba2c2aeb2efe27e402d1dfba1321e256ca30961ef0c: Status 404 returned error can't find the container with id 72e7fd13001c1ad370c8cba2c2aeb2efe27e402d1dfba1321e256ca30961ef0c Jul 22 17:19:18.264260 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:18.264181 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:18.354002 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:18.353961 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:18.914789 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:18.914754 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg" event={"ID":"c546c147-b513-4d3f-ab87-4c0822e21d12","Type":"ContainerStarted","Data":"9012786036b8a8cc759cac5c4dc0ab2910440541572f59d9d38e619c20e6a288"} Jul 22 17:19:18.914947 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:18.914815 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg" event={"ID":"c546c147-b513-4d3f-ab87-4c0822e21d12","Type":"ContainerStarted","Data":"72e7fd13001c1ad370c8cba2c2aeb2efe27e402d1dfba1321e256ca30961ef0c"} Jul 22 17:19:18.953635 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:18.953587 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg" podStartSLOduration=1.953569313 podStartE2EDuration="1.953569313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:19:18.949691243 +0000 UTC m=+2323.926419559" watchObservedRunningTime="2026-07-22 17:19:18.953569313 +0000 UTC m=+2323.930297632" Jul 22 17:19:19.921639 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:19.921603 2557 generic.go:356] "Generic (PLEG): container finished" podID="c546c147-b513-4d3f-ab87-4c0822e21d12" containerID="9012786036b8a8cc759cac5c4dc0ab2910440541572f59d9d38e619c20e6a288" exitCode=0 Jul 22 17:19:19.922067 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:19.921671 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg" event={"ID":"c546c147-b513-4d3f-ab87-4c0822e21d12","Type":"ContainerDied","Data":"9012786036b8a8cc759cac5c4dc0ab2910440541572f59d9d38e619c20e6a288"} Jul 22 17:19:21.058233 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:21.058197 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg" Jul 22 17:19:21.124095 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:21.124065 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c546c147-b513-4d3f-ab87-4c0822e21d12-maas-api-service-ca\") pod \"c546c147-b513-4d3f-ab87-4c0822e21d12\" (UID: \"c546c147-b513-4d3f-ab87-4c0822e21d12\") " Jul 22 17:19:21.124424 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:21.124400 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c546c147-b513-4d3f-ab87-4c0822e21d12-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c546c147-b513-4d3f-ab87-4c0822e21d12" (UID: "c546c147-b513-4d3f-ab87-4c0822e21d12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:19:21.225020 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:21.224931 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c546c147-b513-4d3f-ab87-4c0822e21d12-maas-api-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:19:21.933289 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:21.933203 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg" Jul 22 17:19:21.933289 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:21.933234 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg" event={"ID":"c546c147-b513-4d3f-ab87-4c0822e21d12","Type":"ContainerDied","Data":"72e7fd13001c1ad370c8cba2c2aeb2efe27e402d1dfba1321e256ca30961ef0c"} Jul 22 17:19:21.933289 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:21.933284 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72e7fd13001c1ad370c8cba2c2aeb2efe27e402d1dfba1321e256ca30961ef0c" Jul 22 17:19:28.191998 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.191899 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk"] Jul 22 17:19:28.192462 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.192184 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" podUID="9278e3b6-81cb-4416-b7ff-5ac458ea173f" containerName="maas-api" containerID="cri-o://b2e9ded686fba968ef4e2f8b7f9177a23a848a685f955a9dc0d6f8e3b021f51d" gracePeriod=30 Jul 22 17:19:28.218485 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.218459 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:28.433338 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.433315 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:19:28.502114 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.502071 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9278e3b6-81cb-4416-b7ff-5ac458ea173f-maas-api-tls\") pod \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\" (UID: \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\") " Jul 22 17:19:28.502302 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.502168 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9278e3b6-81cb-4416-b7ff-5ac458ea173f-openshift-service-ca\") pod \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\" (UID: \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\") " Jul 22 17:19:28.502302 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.502247 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfdnx\" (UniqueName: \"kubernetes.io/projected/9278e3b6-81cb-4416-b7ff-5ac458ea173f-kube-api-access-nfdnx\") pod \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\" (UID: \"9278e3b6-81cb-4416-b7ff-5ac458ea173f\") " Jul 22 17:19:28.502513 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.502489 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9278e3b6-81cb-4416-b7ff-5ac458ea173f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9278e3b6-81cb-4416-b7ff-5ac458ea173f" (UID: "9278e3b6-81cb-4416-b7ff-5ac458ea173f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:19:28.504198 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.504175 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9278e3b6-81cb-4416-b7ff-5ac458ea173f-kube-api-access-nfdnx" (OuterVolumeSpecName: "kube-api-access-nfdnx") pod "9278e3b6-81cb-4416-b7ff-5ac458ea173f" (UID: "9278e3b6-81cb-4416-b7ff-5ac458ea173f"). InnerVolumeSpecName "kube-api-access-nfdnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:19:28.504297 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.504278 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9278e3b6-81cb-4416-b7ff-5ac458ea173f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9278e3b6-81cb-4416-b7ff-5ac458ea173f" (UID: "9278e3b6-81cb-4416-b7ff-5ac458ea173f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:19:28.603699 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.603665 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9278e3b6-81cb-4416-b7ff-5ac458ea173f-maas-api-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:19:28.603699 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.603694 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9278e3b6-81cb-4416-b7ff-5ac458ea173f-openshift-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:19:28.603699 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.603703 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfdnx\" (UniqueName: \"kubernetes.io/projected/9278e3b6-81cb-4416-b7ff-5ac458ea173f-kube-api-access-nfdnx\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:19:28.968855 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.968756 2557 generic.go:356] "Generic (PLEG): container finished" podID="9278e3b6-81cb-4416-b7ff-5ac458ea173f" containerID="b2e9ded686fba968ef4e2f8b7f9177a23a848a685f955a9dc0d6f8e3b021f51d" exitCode=0 Jul 22 17:19:28.968855 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.968824 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" event={"ID":"9278e3b6-81cb-4416-b7ff-5ac458ea173f","Type":"ContainerDied","Data":"b2e9ded686fba968ef4e2f8b7f9177a23a848a685f955a9dc0d6f8e3b021f51d"} Jul 22 17:19:28.969100 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.968891 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" Jul 22 17:19:28.969100 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.968919 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk" event={"ID":"9278e3b6-81cb-4416-b7ff-5ac458ea173f","Type":"ContainerDied","Data":"a56396259fe1c9c278c120acf6b414ef7ed0ae5c4bc02cd5ac8f98477c1ac043"} Jul 22 17:19:28.969100 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.968942 2557 scope.go:117] "RemoveContainer" containerID="b2e9ded686fba968ef4e2f8b7f9177a23a848a685f955a9dc0d6f8e3b021f51d" Jul 22 17:19:28.978543 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.978507 2557 scope.go:117] "RemoveContainer" containerID="b2e9ded686fba968ef4e2f8b7f9177a23a848a685f955a9dc0d6f8e3b021f51d" Jul 22 17:19:28.978831 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:19:28.978806 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e9ded686fba968ef4e2f8b7f9177a23a848a685f955a9dc0d6f8e3b021f51d\": container with ID starting with b2e9ded686fba968ef4e2f8b7f9177a23a848a685f955a9dc0d6f8e3b021f51d not found: ID does not exist" containerID="b2e9ded686fba968ef4e2f8b7f9177a23a848a685f955a9dc0d6f8e3b021f51d" Jul 22 17:19:28.978891 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.978841 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e9ded686fba968ef4e2f8b7f9177a23a848a685f955a9dc0d6f8e3b021f51d"} err="failed to get container status \"b2e9ded686fba968ef4e2f8b7f9177a23a848a685f955a9dc0d6f8e3b021f51d\": rpc error: code = NotFound desc = could not find container \"b2e9ded686fba968ef4e2f8b7f9177a23a848a685f955a9dc0d6f8e3b021f51d\": container with ID starting with b2e9ded686fba968ef4e2f8b7f9177a23a848a685f955a9dc0d6f8e3b021f51d not found: ID does not exist" Jul 22 17:19:28.992271 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.992246 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk"] Jul 22 17:19:28.998194 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:28.998164 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f854fd-6fc757dd49-lz2bk"] Jul 22 17:19:29.575844 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:29.575807 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9278e3b6-81cb-4416-b7ff-5ac458ea173f" path="/var/lib/kubelet/pods/9278e3b6-81cb-4416-b7ff-5ac458ea173f/volumes" Jul 22 17:19:32.758544 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:32.758509 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg"] Jul 22 17:19:32.760703 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:32.760674 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f854fd-kbfpg"] Jul 22 17:19:32.761316 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:32.761294 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:33.200088 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:33.199983 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:33.574817 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:33.574786 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c546c147-b513-4d3f-ab87-4c0822e21d12" path="/var/lib/kubelet/pods/c546c147-b513-4d3f-ab87-4c0822e21d12/volumes" Jul 22 17:19:40.296549 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:40.296505 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:40.393297 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:40.393263 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:40.667587 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:40.667507 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:40.913079 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:40.913043 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:41.311571 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:41.311537 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:41.798047 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:41.797980 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:42.212739 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:42.212664 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:45.570310 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.570272 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:45.686723 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.686687 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:45.767464 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.767418 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj"] Jul 22 17:19:45.767959 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.767939 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c546c147-b513-4d3f-ab87-4c0822e21d12" containerName="revoke-keys" Jul 22 17:19:45.768080 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.767961 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c546c147-b513-4d3f-ab87-4c0822e21d12" containerName="revoke-keys" Jul 22 17:19:45.768080 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.767982 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9278e3b6-81cb-4416-b7ff-5ac458ea173f" containerName="maas-api" Jul 22 17:19:45.768080 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.767991 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9278e3b6-81cb-4416-b7ff-5ac458ea173f" containerName="maas-api" Jul 22 17:19:45.768248 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.768098 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9278e3b6-81cb-4416-b7ff-5ac458ea173f" containerName="maas-api" Jul 22 17:19:45.768248 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.768110 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c546c147-b513-4d3f-ab87-4c0822e21d12" containerName="revoke-keys" Jul 22 17:19:45.771311 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.771291 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:19:45.776700 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.776663 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7074f6-serving-cert\"" Jul 22 17:19:45.776820 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.776789 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8sp7j\"" Jul 22 17:19:45.795196 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.795170 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj"] Jul 22 17:19:45.840219 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.840140 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:45.865195 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.865160 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-maas-api-tls\") pod \"maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj\" (UID: \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:19:45.865373 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.865224 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-openshift-service-ca\") pod \"maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj\" (UID: \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:19:45.865373 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.865258 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbbs8\" (UniqueName: \"kubernetes.io/projected/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-kube-api-access-gbbs8\") pod \"maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj\" (UID: \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:19:45.966588 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.966554 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-maas-api-tls\") pod \"maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj\" (UID: \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:19:45.966754 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.966606 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-openshift-service-ca\") pod \"maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj\" (UID: \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:19:45.966754 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.966630 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbbs8\" (UniqueName: \"kubernetes.io/projected/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-kube-api-access-gbbs8\") pod \"maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj\" (UID: \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:19:45.966754 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:19:45.966700 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-serving-cert: secret "maas-api-e2e-api-a-7074f6-serving-cert" not found Jul 22 17:19:45.966880 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:19:45.966773 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-maas-api-tls podName:0d61c0f6-5060-4be7-82b8-fb4e49fb2a46 nodeName:}" failed. No retries permitted until 2026-07-22 17:19:46.466757289 +0000 UTC m=+2351.443485587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-maas-api-tls") pod "maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" (UID: "0d61c0f6-5060-4be7-82b8-fb4e49fb2a46") : secret "maas-api-e2e-api-a-7074f6-serving-cert" not found Jul 22 17:19:45.967276 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.967257 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-openshift-service-ca\") pod \"maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj\" (UID: \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:19:45.979157 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:45.979127 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbbs8\" (UniqueName: \"kubernetes.io/projected/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-kube-api-access-gbbs8\") pod \"maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj\" (UID: \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:19:46.123935 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:46.123846 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:46.471157 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:46.471066 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-maas-api-tls\") pod \"maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj\" (UID: \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:19:46.473490 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:46.473472 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-maas-api-tls\") pod \"maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj\" (UID: \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:19:46.681625 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:46.681590 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:19:46.825257 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:46.825230 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:46.854304 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:19:46.854268 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d61c0f6_5060_4be7_82b8_fb4e49fb2a46.slice/crio-e47f1e27d22a30f346eb4c84ad09c5d5b46d3646871cda409dce2dce6004aa3c WatchSource:0}: Error finding container e47f1e27d22a30f346eb4c84ad09c5d5b46d3646871cda409dce2dce6004aa3c: Status 404 returned error can't find the container with id e47f1e27d22a30f346eb4c84ad09c5d5b46d3646871cda409dce2dce6004aa3c Jul 22 17:19:46.855778 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:46.855756 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 17:19:46.856593 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:46.856378 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj"] Jul 22 17:19:46.944771 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:46.944729 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:47.061448 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:47.061361 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" event={"ID":"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46","Type":"ContainerStarted","Data":"e47f1e27d22a30f346eb4c84ad09c5d5b46d3646871cda409dce2dce6004aa3c"} Jul 22 17:19:48.067889 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:48.067846 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" event={"ID":"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46","Type":"ContainerStarted","Data":"9e0733fa92bce05479e42a2aa046abe46c32c29df497b865cb2714f44cac7119"} Jul 22 17:19:48.068386 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:48.068049 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:19:48.109000 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:48.108947 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" podStartSLOduration=2.531146331 podStartE2EDuration="3.108931153s" totalImagesPullingTime="577.784822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:19:48.108266616 +0000 UTC m=+2353.084994932" watchObservedRunningTime="2026-07-22 17:19:48.108931153 +0000 UTC m=+2353.085659469" Jul 22 17:19:49.980297 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:49.980233 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:50.462456 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:50.462367 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:50.803306 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:50.803274 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:51.862419 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:51.862381 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:53.417421 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:53.417381 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:53.501256 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:53.501225 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:53.505252 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:53.505222 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb"] Jul 22 17:19:53.509393 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:53.509363 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:19:53.512212 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:53.512188 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-050beb-serving-cert\"" Jul 22 17:19:53.520349 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:53.520321 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb"] Jul 22 17:19:53.536673 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:53.536646 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1772c7b-6d28-4973-a774-0314aa509af2-maas-api-tls\") pod \"maas-api-e2e-api-b-050beb-645945699-8lrrb\" (UID: \"e1772c7b-6d28-4973-a774-0314aa509af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:19:53.536863 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:53.536748 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1772c7b-6d28-4973-a774-0314aa509af2-openshift-service-ca\") pod \"maas-api-e2e-api-b-050beb-645945699-8lrrb\" (UID: \"e1772c7b-6d28-4973-a774-0314aa509af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:19:53.536863 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:53.536810 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4pm5\" (UniqueName: \"kubernetes.io/projected/e1772c7b-6d28-4973-a774-0314aa509af2-kube-api-access-n4pm5\") pod \"maas-api-e2e-api-b-050beb-645945699-8lrrb\" (UID: \"e1772c7b-6d28-4973-a774-0314aa509af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:19:53.566263 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:53.566233 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:53.638273 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:53.638240 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1772c7b-6d28-4973-a774-0314aa509af2-openshift-service-ca\") pod \"maas-api-e2e-api-b-050beb-645945699-8lrrb\" (UID: \"e1772c7b-6d28-4973-a774-0314aa509af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:19:53.638556 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:53.638530 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4pm5\" (UniqueName: \"kubernetes.io/projected/e1772c7b-6d28-4973-a774-0314aa509af2-kube-api-access-n4pm5\") pod \"maas-api-e2e-api-b-050beb-645945699-8lrrb\" (UID: \"e1772c7b-6d28-4973-a774-0314aa509af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:19:53.638625 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:53.638608 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1772c7b-6d28-4973-a774-0314aa509af2-maas-api-tls\") pod \"maas-api-e2e-api-b-050beb-645945699-8lrrb\" (UID: \"e1772c7b-6d28-4973-a774-0314aa509af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:19:53.638751 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:19:53.638730 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-serving-cert: secret "maas-api-e2e-api-b-050beb-serving-cert" not found Jul 22 17:19:53.638824 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:19:53.638808 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1772c7b-6d28-4973-a774-0314aa509af2-maas-api-tls podName:e1772c7b-6d28-4973-a774-0314aa509af2 nodeName:}" failed. No retries permitted until 2026-07-22 17:19:54.138791449 +0000 UTC m=+2359.115519756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e1772c7b-6d28-4973-a774-0314aa509af2-maas-api-tls") pod "maas-api-e2e-api-b-050beb-645945699-8lrrb" (UID: "e1772c7b-6d28-4973-a774-0314aa509af2") : secret "maas-api-e2e-api-b-050beb-serving-cert" not found Jul 22 17:19:53.639001 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:53.638969 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1772c7b-6d28-4973-a774-0314aa509af2-openshift-service-ca\") pod \"maas-api-e2e-api-b-050beb-645945699-8lrrb\" (UID: \"e1772c7b-6d28-4973-a774-0314aa509af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:19:53.653672 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:53.653646 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4pm5\" (UniqueName: \"kubernetes.io/projected/e1772c7b-6d28-4973-a774-0314aa509af2-kube-api-access-n4pm5\") pod \"maas-api-e2e-api-b-050beb-645945699-8lrrb\" (UID: \"e1772c7b-6d28-4973-a774-0314aa509af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:19:54.043069 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:54.043010 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:54.078797 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:54.078769 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:19:54.113211 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:54.113176 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:54.143747 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:54.143713 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1772c7b-6d28-4973-a774-0314aa509af2-maas-api-tls\") pod \"maas-api-e2e-api-b-050beb-645945699-8lrrb\" (UID: \"e1772c7b-6d28-4973-a774-0314aa509af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:19:54.146391 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:54.146367 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1772c7b-6d28-4973-a774-0314aa509af2-maas-api-tls\") pod \"maas-api-e2e-api-b-050beb-645945699-8lrrb\" (UID: \"e1772c7b-6d28-4973-a774-0314aa509af2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:19:54.175563 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:54.175516 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:19:54.421384 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:54.421301 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:19:54.571589 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:54.571555 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb"] Jul 22 17:19:54.572124 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:19:54.572097 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1772c7b_6d28_4973_a774_0314aa509af2.slice/crio-04d4813ef27d13887739e7a53d48e8f1499bc5ac4b86bcaa2d0398c3b2f49603 WatchSource:0}: Error finding container 04d4813ef27d13887739e7a53d48e8f1499bc5ac4b86bcaa2d0398c3b2f49603: Status 404 returned error can't find the container with id 04d4813ef27d13887739e7a53d48e8f1499bc5ac4b86bcaa2d0398c3b2f49603 Jul 22 17:19:55.105914 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:55.105881 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" event={"ID":"e1772c7b-6d28-4973-a774-0314aa509af2","Type":"ContainerStarted","Data":"5dde02078380c03493489bb9bdf38310964bf77a45b8ab1c839f0c3cce00c548"} Jul 22 17:19:55.106099 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:55.105930 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" event={"ID":"e1772c7b-6d28-4973-a774-0314aa509af2","Type":"ContainerStarted","Data":"04d4813ef27d13887739e7a53d48e8f1499bc5ac4b86bcaa2d0398c3b2f49603"} Jul 22 17:19:55.106099 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:55.105957 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:19:55.129979 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:55.129933 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" podStartSLOduration=1.784593223 podStartE2EDuration="2.129916979s" totalImagesPullingTime="345.323756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:19:55.12970936 +0000 UTC m=+2360.106437676" watchObservedRunningTime="2026-07-22 17:19:55.129916979 +0000 UTC m=+2360.106645296" Jul 22 17:19:57.969373 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:19:57.969337 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:01.116295 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:01.116253 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:20:06.251853 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:06.251810 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt"] Jul 22 17:20:06.256007 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:06.255984 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt" Jul 22 17:20:06.259320 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:06.259300 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vln7b\"" Jul 22 17:20:06.270122 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:06.270097 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt"] Jul 22 17:20:06.356982 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:06.356942 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37e95e40-95bf-4d19-aa43-16aa1b7bae24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt\" (UID: \"37e95e40-95bf-4d19-aa43-16aa1b7bae24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt" Jul 22 17:20:06.458510 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:06.458473 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37e95e40-95bf-4d19-aa43-16aa1b7bae24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt\" (UID: \"37e95e40-95bf-4d19-aa43-16aa1b7bae24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt" Jul 22 17:20:06.459139 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:06.459120 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37e95e40-95bf-4d19-aa43-16aa1b7bae24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt\" (UID: \"37e95e40-95bf-4d19-aa43-16aa1b7bae24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt" Jul 22 17:20:06.565700 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:06.565608 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt" Jul 22 17:20:06.707806 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:06.707775 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt"] Jul 22 17:20:06.710106 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:20:06.710077 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e95e40_95bf_4d19_aa43_16aa1b7bae24.slice/crio-ff97b4236aaa0de242600d946677a14b8b1061d70a583fecef246fb400781b26 WatchSource:0}: Error finding container ff97b4236aaa0de242600d946677a14b8b1061d70a583fecef246fb400781b26: Status 404 returned error can't find the container with id ff97b4236aaa0de242600d946677a14b8b1061d70a583fecef246fb400781b26 Jul 22 17:20:06.916907 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:06.916826 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:07.019407 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:07.019372 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:07.165986 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:07.165947 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt" event={"ID":"37e95e40-95bf-4d19-aa43-16aa1b7bae24","Type":"ContainerStarted","Data":"b0b8e938796b4f75ed6fcd311d161df87db172eb83c88d4882abfe40321c4a47"} Jul 22 17:20:07.166197 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:07.166012 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt" event={"ID":"37e95e40-95bf-4d19-aa43-16aa1b7bae24","Type":"ContainerStarted","Data":"ff97b4236aaa0de242600d946677a14b8b1061d70a583fecef246fb400781b26"} Jul 22 17:20:07.184265 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:07.184160 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt" podStartSLOduration=1.184143078 podStartE2EDuration="1.184143078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:20:07.182255514 +0000 UTC m=+2372.158983832" watchObservedRunningTime="2026-07-22 17:20:07.184143078 +0000 UTC m=+2372.160871409" Jul 22 17:20:08.173392 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:08.173358 2557 generic.go:356] "Generic (PLEG): container finished" podID="37e95e40-95bf-4d19-aa43-16aa1b7bae24" containerID="b0b8e938796b4f75ed6fcd311d161df87db172eb83c88d4882abfe40321c4a47" exitCode=0 Jul 22 17:20:08.173784 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:08.173423 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt" event={"ID":"37e95e40-95bf-4d19-aa43-16aa1b7bae24","Type":"ContainerDied","Data":"b0b8e938796b4f75ed6fcd311d161df87db172eb83c88d4882abfe40321c4a47"} Jul 22 17:20:09.331489 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:09.331463 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt" Jul 22 17:20:09.485802 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:09.485765 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37e95e40-95bf-4d19-aa43-16aa1b7bae24-maas-api-service-ca\") pod \"37e95e40-95bf-4d19-aa43-16aa1b7bae24\" (UID: \"37e95e40-95bf-4d19-aa43-16aa1b7bae24\") " Jul 22 17:20:09.486148 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:09.486121 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e95e40-95bf-4d19-aa43-16aa1b7bae24-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37e95e40-95bf-4d19-aa43-16aa1b7bae24" (UID: "37e95e40-95bf-4d19-aa43-16aa1b7bae24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:20:09.586536 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:09.586505 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37e95e40-95bf-4d19-aa43-16aa1b7bae24-maas-api-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:20:10.184881 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:10.184852 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt" Jul 22 17:20:10.185072 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:10.184881 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt" event={"ID":"37e95e40-95bf-4d19-aa43-16aa1b7bae24","Type":"ContainerDied","Data":"ff97b4236aaa0de242600d946677a14b8b1061d70a583fecef246fb400781b26"} Jul 22 17:20:10.185072 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:10.184934 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff97b4236aaa0de242600d946677a14b8b1061d70a583fecef246fb400781b26" Jul 22 17:20:14.884797 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:14.884762 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:14.989607 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:14.989570 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:16.965330 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:16.965293 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:17.063084 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:17.063047 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:22.233442 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:22.233411 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj"] Jul 22 17:20:22.233876 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:22.233632 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" podUID="0d61c0f6-5060-4be7-82b8-fb4e49fb2a46" containerName="maas-api" containerID="cri-o://9e0733fa92bce05479e42a2aa046abe46c32c29df497b865cb2714f44cac7119" gracePeriod=30 Jul 22 17:20:22.264006 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:22.263980 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:22.483654 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:22.483590 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:20:22.612845 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:22.612812 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbbs8\" (UniqueName: \"kubernetes.io/projected/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-kube-api-access-gbbs8\") pod \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\" (UID: \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\") " Jul 22 17:20:22.612845 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:22.612850 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-maas-api-tls\") pod \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\" (UID: \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\") " Jul 22 17:20:22.613103 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:22.612878 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-openshift-service-ca\") pod \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\" (UID: \"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46\") " Jul 22 17:20:22.613361 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:22.613334 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0d61c0f6-5060-4be7-82b8-fb4e49fb2a46" (UID: "0d61c0f6-5060-4be7-82b8-fb4e49fb2a46"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:20:22.615050 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:22.615008 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-kube-api-access-gbbs8" (OuterVolumeSpecName: "kube-api-access-gbbs8") pod "0d61c0f6-5060-4be7-82b8-fb4e49fb2a46" (UID: "0d61c0f6-5060-4be7-82b8-fb4e49fb2a46"). InnerVolumeSpecName "kube-api-access-gbbs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:20:22.615050 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:22.615008 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0d61c0f6-5060-4be7-82b8-fb4e49fb2a46" (UID: "0d61c0f6-5060-4be7-82b8-fb4e49fb2a46"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:20:22.714763 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:22.714720 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbbs8\" (UniqueName: \"kubernetes.io/projected/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-kube-api-access-gbbs8\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:20:22.714763 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:22.714757 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-maas-api-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:20:22.714763 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:22.714767 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46-openshift-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:20:23.253744 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:23.253704 2557 generic.go:356] "Generic (PLEG): container finished" podID="0d61c0f6-5060-4be7-82b8-fb4e49fb2a46" containerID="9e0733fa92bce05479e42a2aa046abe46c32c29df497b865cb2714f44cac7119" exitCode=0 Jul 22 17:20:23.253744 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:23.253746 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" Jul 22 17:20:23.254237 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:23.253793 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" event={"ID":"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46","Type":"ContainerDied","Data":"9e0733fa92bce05479e42a2aa046abe46c32c29df497b865cb2714f44cac7119"} Jul 22 17:20:23.254237 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:23.253843 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj" event={"ID":"0d61c0f6-5060-4be7-82b8-fb4e49fb2a46","Type":"ContainerDied","Data":"e47f1e27d22a30f346eb4c84ad09c5d5b46d3646871cda409dce2dce6004aa3c"} Jul 22 17:20:23.254237 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:23.253867 2557 scope.go:117] "RemoveContainer" containerID="9e0733fa92bce05479e42a2aa046abe46c32c29df497b865cb2714f44cac7119" Jul 22 17:20:23.263061 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:23.263040 2557 scope.go:117] "RemoveContainer" containerID="9e0733fa92bce05479e42a2aa046abe46c32c29df497b865cb2714f44cac7119" Jul 22 17:20:23.263324 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:20:23.263306 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0733fa92bce05479e42a2aa046abe46c32c29df497b865cb2714f44cac7119\": container with ID starting with 9e0733fa92bce05479e42a2aa046abe46c32c29df497b865cb2714f44cac7119 not found: ID does not exist" containerID="9e0733fa92bce05479e42a2aa046abe46c32c29df497b865cb2714f44cac7119" Jul 22 17:20:23.263407 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:23.263336 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0733fa92bce05479e42a2aa046abe46c32c29df497b865cb2714f44cac7119"} err="failed to get container status \"9e0733fa92bce05479e42a2aa046abe46c32c29df497b865cb2714f44cac7119\": rpc error: code = NotFound desc = could not find container \"9e0733fa92bce05479e42a2aa046abe46c32c29df497b865cb2714f44cac7119\": container with ID starting with 9e0733fa92bce05479e42a2aa046abe46c32c29df497b865cb2714f44cac7119 not found: ID does not exist" Jul 22 17:20:23.285555 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:23.285530 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj"] Jul 22 17:20:23.304128 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:23.304094 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7074f6-5b9578f68b-4ffhj"] Jul 22 17:20:23.575908 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:23.575828 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d61c0f6-5060-4be7-82b8-fb4e49fb2a46" path="/var/lib/kubelet/pods/0d61c0f6-5060-4be7-82b8-fb4e49fb2a46/volumes" Jul 22 17:20:26.370961 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:26.370929 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:26.397385 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:26.397354 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt"] Jul 22 17:20:26.404807 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:26.404781 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7074f6-hbgvt"] Jul 22 17:20:26.797324 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:26.797278 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:27.574840 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:27.574808 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37e95e40-95bf-4d19-aa43-16aa1b7bae24" path="/var/lib/kubelet/pods/37e95e40-95bf-4d19-aa43-16aa1b7bae24/volumes" Jul 22 17:20:33.473349 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:33.473310 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:33.573788 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:33.573751 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:33.728406 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:33.728371 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:33.955185 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:33.955155 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:34.075871 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.075786 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-050beb-44mk2"] Jul 22 17:20:34.076209 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.076196 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37e95e40-95bf-4d19-aa43-16aa1b7bae24" containerName="revoke-keys" Jul 22 17:20:34.076280 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.076210 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="37e95e40-95bf-4d19-aa43-16aa1b7bae24" containerName="revoke-keys" Jul 22 17:20:34.076280 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.076229 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d61c0f6-5060-4be7-82b8-fb4e49fb2a46" containerName="maas-api" Jul 22 17:20:34.076280 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.076235 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d61c0f6-5060-4be7-82b8-fb4e49fb2a46" containerName="maas-api" Jul 22 17:20:34.076399 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.076298 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d61c0f6-5060-4be7-82b8-fb4e49fb2a46" containerName="maas-api" Jul 22 17:20:34.076399 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.076308 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="37e95e40-95bf-4d19-aa43-16aa1b7bae24" containerName="revoke-keys" Jul 22 17:20:34.079429 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.079410 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-050beb-44mk2" Jul 22 17:20:34.082110 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.082086 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vln7b\"" Jul 22 17:20:34.087132 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.087009 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-050beb-44mk2"] Jul 22 17:20:34.123805 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.123775 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f2e0db-dfbd-45ad-a374-f312b6bb2d75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-050beb-44mk2\" (UID: \"a8f2e0db-dfbd-45ad-a374-f312b6bb2d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-050beb-44mk2" Jul 22 17:20:34.225519 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.225476 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f2e0db-dfbd-45ad-a374-f312b6bb2d75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-050beb-44mk2\" (UID: \"a8f2e0db-dfbd-45ad-a374-f312b6bb2d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-050beb-44mk2" Jul 22 17:20:34.226149 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.226123 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f2e0db-dfbd-45ad-a374-f312b6bb2d75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-050beb-44mk2\" (UID: \"a8f2e0db-dfbd-45ad-a374-f312b6bb2d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-050beb-44mk2" Jul 22 17:20:34.391725 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.391635 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-050beb-44mk2" Jul 22 17:20:34.576015 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.575989 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-050beb-44mk2"] Jul 22 17:20:34.578771 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:20:34.578745 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f2e0db_dfbd_45ad_a374_f312b6bb2d75.slice/crio-b5afbb0b6d6dc7782f29fb88e8ff4b37c62d1f77de10642a3bb263dae9592db4 WatchSource:0}: Error finding container b5afbb0b6d6dc7782f29fb88e8ff4b37c62d1f77de10642a3bb263dae9592db4: Status 404 returned error can't find the container with id b5afbb0b6d6dc7782f29fb88e8ff4b37c62d1f77de10642a3bb263dae9592db4 Jul 22 17:20:34.754483 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.754410 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:34.847126 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:34.847089 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:35.319917 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:35.319877 2557 generic.go:356] "Generic (PLEG): container finished" podID="a8f2e0db-dfbd-45ad-a374-f312b6bb2d75" containerID="b998d1ebe7b31ff5c7c4fdfa70e32ef9ebd1dd95c5102d3fab6bdfef202c636e" exitCode=0 Jul 22 17:20:35.320067 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:35.319960 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-050beb-44mk2" event={"ID":"a8f2e0db-dfbd-45ad-a374-f312b6bb2d75","Type":"ContainerDied","Data":"b998d1ebe7b31ff5c7c4fdfa70e32ef9ebd1dd95c5102d3fab6bdfef202c636e"} Jul 22 17:20:35.320067 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:35.320010 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-050beb-44mk2" event={"ID":"a8f2e0db-dfbd-45ad-a374-f312b6bb2d75","Type":"ContainerStarted","Data":"b5afbb0b6d6dc7782f29fb88e8ff4b37c62d1f77de10642a3bb263dae9592db4"} Jul 22 17:20:35.801661 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:35.801629 2557 scope.go:117] "RemoveContainer" containerID="b99e7b09e1e9e84ea1fc99acb1502eb8d5fe39e64b899d0dce6aa483a8e083bf" Jul 22 17:20:36.458513 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:36.458488 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-050beb-44mk2" Jul 22 17:20:36.548413 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:36.548378 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f2e0db-dfbd-45ad-a374-f312b6bb2d75-maas-api-service-ca\") pod \"a8f2e0db-dfbd-45ad-a374-f312b6bb2d75\" (UID: \"a8f2e0db-dfbd-45ad-a374-f312b6bb2d75\") " Jul 22 17:20:36.548746 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:36.548722 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f2e0db-dfbd-45ad-a374-f312b6bb2d75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a8f2e0db-dfbd-45ad-a374-f312b6bb2d75" (UID: "a8f2e0db-dfbd-45ad-a374-f312b6bb2d75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:20:36.649360 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:36.649281 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f2e0db-dfbd-45ad-a374-f312b6bb2d75-maas-api-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:20:37.331352 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:37.331312 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-050beb-44mk2" event={"ID":"a8f2e0db-dfbd-45ad-a374-f312b6bb2d75","Type":"ContainerDied","Data":"b5afbb0b6d6dc7782f29fb88e8ff4b37c62d1f77de10642a3bb263dae9592db4"} Jul 22 17:20:37.331352 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:37.331343 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-050beb-44mk2" Jul 22 17:20:37.331780 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:37.331375 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5afbb0b6d6dc7782f29fb88e8ff4b37c62d1f77de10642a3bb263dae9592db4" Jul 22 17:20:49.795120 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:49.795087 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb"] Jul 22 17:20:49.795593 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:49.795333 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" podUID="e1772c7b-6d28-4973-a774-0314aa509af2" containerName="maas-api" containerID="cri-o://5dde02078380c03493489bb9bdf38310964bf77a45b8ab1c839f0c3cce00c548" gracePeriod=30 Jul 22 17:20:49.829847 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:49.829820 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:50.043919 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.043891 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:20:50.172346 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.172261 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1772c7b-6d28-4973-a774-0314aa509af2-maas-api-tls\") pod \"e1772c7b-6d28-4973-a774-0314aa509af2\" (UID: \"e1772c7b-6d28-4973-a774-0314aa509af2\") " Jul 22 17:20:50.172346 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.172301 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1772c7b-6d28-4973-a774-0314aa509af2-openshift-service-ca\") pod \"e1772c7b-6d28-4973-a774-0314aa509af2\" (UID: \"e1772c7b-6d28-4973-a774-0314aa509af2\") " Jul 22 17:20:50.172566 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.172349 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4pm5\" (UniqueName: \"kubernetes.io/projected/e1772c7b-6d28-4973-a774-0314aa509af2-kube-api-access-n4pm5\") pod \"e1772c7b-6d28-4973-a774-0314aa509af2\" (UID: \"e1772c7b-6d28-4973-a774-0314aa509af2\") " Jul 22 17:20:50.172669 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.172644 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1772c7b-6d28-4973-a774-0314aa509af2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e1772c7b-6d28-4973-a774-0314aa509af2" (UID: "e1772c7b-6d28-4973-a774-0314aa509af2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:20:50.174350 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.174319 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1772c7b-6d28-4973-a774-0314aa509af2-kube-api-access-n4pm5" (OuterVolumeSpecName: "kube-api-access-n4pm5") pod "e1772c7b-6d28-4973-a774-0314aa509af2" (UID: "e1772c7b-6d28-4973-a774-0314aa509af2"). InnerVolumeSpecName "kube-api-access-n4pm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:20:50.174471 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.174379 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1772c7b-6d28-4973-a774-0314aa509af2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e1772c7b-6d28-4973-a774-0314aa509af2" (UID: "e1772c7b-6d28-4973-a774-0314aa509af2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:20:50.274068 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.274009 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4pm5\" (UniqueName: \"kubernetes.io/projected/e1772c7b-6d28-4973-a774-0314aa509af2-kube-api-access-n4pm5\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:20:50.274068 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.274063 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1772c7b-6d28-4973-a774-0314aa509af2-maas-api-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:20:50.274068 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.274072 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1772c7b-6d28-4973-a774-0314aa509af2-openshift-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:20:50.403114 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.403078 2557 generic.go:356] "Generic (PLEG): container finished" podID="e1772c7b-6d28-4973-a774-0314aa509af2" containerID="5dde02078380c03493489bb9bdf38310964bf77a45b8ab1c839f0c3cce00c548" exitCode=0 Jul 22 17:20:50.403276 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.403126 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" Jul 22 17:20:50.403276 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.403170 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" event={"ID":"e1772c7b-6d28-4973-a774-0314aa509af2","Type":"ContainerDied","Data":"5dde02078380c03493489bb9bdf38310964bf77a45b8ab1c839f0c3cce00c548"} Jul 22 17:20:50.403276 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.403223 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb" event={"ID":"e1772c7b-6d28-4973-a774-0314aa509af2","Type":"ContainerDied","Data":"04d4813ef27d13887739e7a53d48e8f1499bc5ac4b86bcaa2d0398c3b2f49603"} Jul 22 17:20:50.403276 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.403249 2557 scope.go:117] "RemoveContainer" containerID="5dde02078380c03493489bb9bdf38310964bf77a45b8ab1c839f0c3cce00c548" Jul 22 17:20:50.412243 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.412232 2557 scope.go:117] "RemoveContainer" containerID="5dde02078380c03493489bb9bdf38310964bf77a45b8ab1c839f0c3cce00c548" Jul 22 17:20:50.412498 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:20:50.412478 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dde02078380c03493489bb9bdf38310964bf77a45b8ab1c839f0c3cce00c548\": container with ID starting with 5dde02078380c03493489bb9bdf38310964bf77a45b8ab1c839f0c3cce00c548 not found: ID does not exist" containerID="5dde02078380c03493489bb9bdf38310964bf77a45b8ab1c839f0c3cce00c548" Jul 22 17:20:50.412547 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.412507 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dde02078380c03493489bb9bdf38310964bf77a45b8ab1c839f0c3cce00c548"} err="failed to get container status \"5dde02078380c03493489bb9bdf38310964bf77a45b8ab1c839f0c3cce00c548\": rpc error: code = NotFound desc = could not find container \"5dde02078380c03493489bb9bdf38310964bf77a45b8ab1c839f0c3cce00c548\": container with ID starting with 5dde02078380c03493489bb9bdf38310964bf77a45b8ab1c839f0c3cce00c548 not found: ID does not exist" Jul 22 17:20:50.434732 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.434648 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb"] Jul 22 17:20:50.438604 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:50.438577 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-050beb-645945699-8lrrb"] Jul 22 17:20:51.575617 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:51.575581 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1772c7b-6d28-4973-a774-0314aa509af2" path="/var/lib/kubelet/pods/e1772c7b-6d28-4973-a774-0314aa509af2/volumes" Jul 22 17:20:51.947547 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:51.947463 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:54.165195 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:54.165155 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:54.171001 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:54.170973 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-050beb-44mk2"] Jul 22 17:20:54.174617 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:54.174594 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-050beb-44mk2"] Jul 22 17:20:54.585102 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:54.585066 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:20:55.575401 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:20:55.575355 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8f2e0db-dfbd-45ad-a374-f312b6bb2d75" path="/var/lib/kubelet/pods/a8f2e0db-dfbd-45ad-a374-f312b6bb2d75/volumes" Jul 22 17:21:00.779536 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:00.779425 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:00.879527 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:00.879493 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:01.070440 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:01.070354 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:01.334937 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:01.334848 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:01.679082 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:01.678949 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:01.993740 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:01.993704 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:02.282015 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:02.281934 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:04.740685 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:04.740645 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:04.835991 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:04.835955 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:04.929249 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:04.929216 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq"] Jul 22 17:21:04.929625 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:04.929612 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f2e0db-dfbd-45ad-a374-f312b6bb2d75" containerName="revoke-keys" Jul 22 17:21:04.929670 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:04.929629 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f2e0db-dfbd-45ad-a374-f312b6bb2d75" containerName="revoke-keys" Jul 22 17:21:04.929670 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:04.929644 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1772c7b-6d28-4973-a774-0314aa509af2" containerName="maas-api" Jul 22 17:21:04.929670 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:04.929650 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1772c7b-6d28-4973-a774-0314aa509af2" containerName="maas-api" Jul 22 17:21:04.929765 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:04.929729 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1772c7b-6d28-4973-a774-0314aa509af2" containerName="maas-api" Jul 22 17:21:04.929765 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:04.929741 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f2e0db-dfbd-45ad-a374-f312b6bb2d75" containerName="revoke-keys" Jul 22 17:21:04.934385 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:04.934365 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:21:04.937076 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:04.937053 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8sp7j\"" Jul 22 17:21:04.937205 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:04.937166 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-629c13-serving-cert\"" Jul 22 17:21:04.944646 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:04.944621 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq"] Jul 22 17:21:04.979448 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:04.979399 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:05.105452 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:05.105367 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9rwx\" (UniqueName: \"kubernetes.io/projected/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-kube-api-access-m9rwx\") pod \"maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq\" (UID: \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:21:05.105452 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:05.105418 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-maas-api-tls\") pod \"maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq\" (UID: \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:21:05.105452 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:05.105439 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq\" (UID: \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:21:05.206194 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:05.206162 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9rwx\" (UniqueName: \"kubernetes.io/projected/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-kube-api-access-m9rwx\") pod \"maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq\" (UID: \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:21:05.206368 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:05.206205 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-maas-api-tls\") pod \"maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq\" (UID: \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:21:05.206368 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:05.206223 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq\" (UID: \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:21:05.206368 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:21:05.206332 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-serving-cert: secret "maas-api-e2e-shared-a-629c13-serving-cert" not found Jul 22 17:21:05.206476 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:21:05.206396 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-maas-api-tls podName:3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5 nodeName:}" failed. No retries permitted until 2026-07-22 17:21:05.706377025 +0000 UTC m=+2430.683105320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-maas-api-tls") pod "maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" (UID: "3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5") : secret "maas-api-e2e-shared-a-629c13-serving-cert" not found Jul 22 17:21:05.206806 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:05.206790 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq\" (UID: \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:21:05.223372 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:05.223337 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9rwx\" (UniqueName: \"kubernetes.io/projected/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-kube-api-access-m9rwx\") pod \"maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq\" (UID: \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:21:05.464723 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:05.464637 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:05.647853 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:05.647823 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:05.718914 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:05.712982 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-maas-api-tls\") pod \"maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq\" (UID: \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:21:05.720730 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:05.720703 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-maas-api-tls\") pod \"maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq\" (UID: \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:21:05.748618 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:05.748586 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:05.846730 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:05.846689 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:21:05.991808 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:05.991776 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq"] Jul 22 17:21:05.993724 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:21:05.993693 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb90ea1_f0d1_4ab0_bcad_5dd2a481c7a5.slice/crio-918cdea168360c4ee02ce2d63d745948a1869f5f0ba78fd6554e44a98e312554 WatchSource:0}: Error finding container 918cdea168360c4ee02ce2d63d745948a1869f5f0ba78fd6554e44a98e312554: Status 404 returned error can't find the container with id 918cdea168360c4ee02ce2d63d745948a1869f5f0ba78fd6554e44a98e312554 Jul 22 17:21:06.484379 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:06.484344 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" event={"ID":"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5","Type":"ContainerStarted","Data":"e6da79149f7869eee2b89647168516e7b899aca3c9467f26a2b0eb9150acc13e"} Jul 22 17:21:06.484570 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:06.484393 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" event={"ID":"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5","Type":"ContainerStarted","Data":"918cdea168360c4ee02ce2d63d745948a1869f5f0ba78fd6554e44a98e312554"} Jul 22 17:21:06.484570 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:06.484447 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:21:06.511826 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:06.511775 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" podStartSLOduration=2.131722912 podStartE2EDuration="2.511758381s" totalImagesPullingTime="380.035469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:21:06.510918389 +0000 UTC m=+2431.487646706" watchObservedRunningTime="2026-07-22 17:21:06.511758381 +0000 UTC m=+2431.488486697" Jul 22 17:21:09.261542 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:09.261508 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:09.628540 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:09.628455 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:09.948071 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:09.947958 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:10.112660 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:10.112629 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:11.042567 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:11.042535 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:11.144580 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:11.144547 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:12.494675 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:12.494646 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:21:12.850068 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:12.849952 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:12.936372 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:12.936338 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:13.114722 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.114634 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb"] Jul 22 17:21:13.118363 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.118343 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:21:13.121742 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.121719 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-05e61d-serving-cert\"" Jul 22 17:21:13.129838 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.129814 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb"] Jul 22 17:21:13.150228 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.150197 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:13.182435 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.182406 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7gn9\" (UniqueName: \"kubernetes.io/projected/37d986d8-ff16-44bd-a924-9180a88bc7e6-kube-api-access-c7gn9\") pod \"maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb\" (UID: \"37d986d8-ff16-44bd-a924-9180a88bc7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:21:13.182558 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.182443 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d986d8-ff16-44bd-a924-9180a88bc7e6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb\" (UID: \"37d986d8-ff16-44bd-a924-9180a88bc7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:21:13.182600 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.182556 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37d986d8-ff16-44bd-a924-9180a88bc7e6-maas-api-tls\") pod \"maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb\" (UID: \"37d986d8-ff16-44bd-a924-9180a88bc7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:21:13.283485 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.283448 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37d986d8-ff16-44bd-a924-9180a88bc7e6-maas-api-tls\") pod \"maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb\" (UID: \"37d986d8-ff16-44bd-a924-9180a88bc7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:21:13.283485 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.283488 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7gn9\" (UniqueName: \"kubernetes.io/projected/37d986d8-ff16-44bd-a924-9180a88bc7e6-kube-api-access-c7gn9\") pod \"maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb\" (UID: \"37d986d8-ff16-44bd-a924-9180a88bc7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:21:13.283690 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.283513 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d986d8-ff16-44bd-a924-9180a88bc7e6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb\" (UID: \"37d986d8-ff16-44bd-a924-9180a88bc7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:21:13.283733 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:21:13.283710 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-serving-cert: secret "maas-api-e2e-shared-b-05e61d-serving-cert" not found Jul 22 17:21:13.283788 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:21:13.283778 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d986d8-ff16-44bd-a924-9180a88bc7e6-maas-api-tls podName:37d986d8-ff16-44bd-a924-9180a88bc7e6 nodeName:}" failed. No retries permitted until 2026-07-22 17:21:13.783762234 +0000 UTC m=+2438.760490529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/37d986d8-ff16-44bd-a924-9180a88bc7e6-maas-api-tls") pod "maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" (UID: "37d986d8-ff16-44bd-a924-9180a88bc7e6") : secret "maas-api-e2e-shared-b-05e61d-serving-cert" not found Jul 22 17:21:13.284137 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.284121 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d986d8-ff16-44bd-a924-9180a88bc7e6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb\" (UID: \"37d986d8-ff16-44bd-a924-9180a88bc7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:21:13.294446 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.294419 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7gn9\" (UniqueName: \"kubernetes.io/projected/37d986d8-ff16-44bd-a924-9180a88bc7e6-kube-api-access-c7gn9\") pod \"maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb\" (UID: \"37d986d8-ff16-44bd-a924-9180a88bc7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:21:13.390164 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.390081 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:13.779839 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.779803 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:13.788152 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.788124 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37d986d8-ff16-44bd-a924-9180a88bc7e6-maas-api-tls\") pod \"maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb\" (UID: \"37d986d8-ff16-44bd-a924-9180a88bc7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:21:13.790571 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.790547 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37d986d8-ff16-44bd-a924-9180a88bc7e6-maas-api-tls\") pod \"maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb\" (UID: \"37d986d8-ff16-44bd-a924-9180a88bc7e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:21:13.881579 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:13.881546 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:14.030088 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:14.029986 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:21:14.168122 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:14.168094 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb"] Jul 22 17:21:14.169975 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:21:14.169944 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d986d8_ff16_44bd_a924_9180a88bc7e6.slice/crio-309c1f00505dc51e930199e42eeecdf2eddf7c5e50df5680702b5b9e4aa4de4b WatchSource:0}: Error finding container 309c1f00505dc51e930199e42eeecdf2eddf7c5e50df5680702b5b9e4aa4de4b: Status 404 returned error can't find the container with id 309c1f00505dc51e930199e42eeecdf2eddf7c5e50df5680702b5b9e4aa4de4b Jul 22 17:21:14.526793 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:14.526769 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" event={"ID":"37d986d8-ff16-44bd-a924-9180a88bc7e6","Type":"ContainerStarted","Data":"309c1f00505dc51e930199e42eeecdf2eddf7c5e50df5680702b5b9e4aa4de4b"} Jul 22 17:21:15.533658 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:15.533615 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" event={"ID":"37d986d8-ff16-44bd-a924-9180a88bc7e6","Type":"ContainerStarted","Data":"636f866c4ab82ba63685e0e3b99225abe8c5150c6881869ec6211d829b00b8bb"} Jul 22 17:21:15.534014 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:15.533796 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:21:15.552573 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:15.552531 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" podStartSLOduration=2.259385127 podStartE2EDuration="2.552518077s" totalImagesPullingTime="293.13295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:21:15.550269465 +0000 UTC m=+2440.526997787" watchObservedRunningTime="2026-07-22 17:21:15.552518077 +0000 UTC m=+2440.529246453" Jul 22 17:21:17.435648 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:17.435614 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:21.545281 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:21.545250 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:21:26.463969 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:26.463924 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:26.564824 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:26.564789 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:32.752401 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:32.752359 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:35.621355 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:35.621314 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:35.723419 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:35.723382 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:35.827839 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:35.827809 2557 scope.go:117] "RemoveContainer" containerID="4ba4167277fa17c57e19d47c4b2eefb2e562630190ad0ccd2ce3885eb7ef3f2d" Jul 22 17:21:35.836254 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:35.836233 2557 scope.go:117] "RemoveContainer" containerID="5cba3aedecf348c945d8a5e3f5ae34f8427eea6e23067a752a3d083b4b79c830" Jul 22 17:21:37.708822 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:37.708790 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:21:37.810084 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:21:37.810047 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:22:10.029742 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:22:10.029703 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:22:14.218259 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:22:14.218217 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:24:18.608695 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:24:18.608655 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:24:20.255395 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:24:20.255353 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:24:27.520721 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:24:27.520686 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:24:43.558242 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:24:43.558197 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:24:43.655402 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:24:43.655365 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:24:45.521838 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:24:45.521804 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:25:25.402518 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:25:25.402474 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:25:26.334248 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:25:26.334205 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:25:35.893876 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:25:35.893844 2557 scope.go:117] "RemoveContainer" containerID="d9dfe5d9a75ea7f3fecabd935cd320b5c89302af40aebd445b0ee8467c69d408" Jul 22 17:25:35.902372 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:25:35.902348 2557 scope.go:117] "RemoveContainer" containerID="9012786036b8a8cc759cac5c4dc0ab2910440541572f59d9d38e619c20e6a288" Jul 22 17:25:38.729541 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:25:38.729488 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:26:11.953076 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:26:11.953012 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:26:12.053381 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:26:12.053336 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:26:20.226053 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:26:20.225985 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:26:27.270381 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:26:27.270335 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:26:27.370173 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:26:27.370126 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:26:35.925420 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:26:35.925388 2557 scope.go:117] "RemoveContainer" containerID="b998d1ebe7b31ff5c7c4fdfa70e32ef9ebd1dd95c5102d3fab6bdfef202c636e" Jul 22 17:26:35.934094 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:26:35.934073 2557 scope.go:117] "RemoveContainer" containerID="b0b8e938796b4f75ed6fcd311d161df87db172eb83c88d4882abfe40321c4a47" Jul 22 17:26:36.386688 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:26:36.386657 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:26:36.486043 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:26:36.485977 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:26:50.113795 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:26:50.113757 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:27:05.262403 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:05.262300 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:27:06.237751 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:06.237698 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:27:25.445888 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:25.445849 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g"] Jul 22 17:27:25.450211 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:25.450192 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g" Jul 22 17:27:25.452968 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:25.452943 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vln7b\"" Jul 22 17:27:25.457013 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:25.456986 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g"] Jul 22 17:27:25.578765 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:25.578724 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-629c13-95c4g\" (UID: \"2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g" Jul 22 17:27:25.679630 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:25.679595 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-629c13-95c4g\" (UID: \"2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g" Jul 22 17:27:25.680228 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:25.680210 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-629c13-95c4g\" (UID: \"2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g" Jul 22 17:27:25.764237 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:25.764202 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g" Jul 22 17:27:25.901786 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:25.901763 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g"] Jul 22 17:27:25.903853 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:27:25.903815 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0461fb_4bcc_4c9b_a8b7_aaaecd0502f3.slice/crio-eb8ec46b0b5420703f56f96aeec6fd3a12ff459a520a184b6455d60e38bd646a WatchSource:0}: Error finding container eb8ec46b0b5420703f56f96aeec6fd3a12ff459a520a184b6455d60e38bd646a: Status 404 returned error can't find the container with id eb8ec46b0b5420703f56f96aeec6fd3a12ff459a520a184b6455d60e38bd646a Jul 22 17:27:26.053687 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:26.053603 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:27:26.154142 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:26.154098 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:27:26.394995 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:26.394898 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g" event={"ID":"2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3","Type":"ContainerStarted","Data":"05192aea73954e684c246c8d7d69c1b48b28284d1ca64afe1e238293cee9f68d"} Jul 22 17:27:26.394995 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:26.394961 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g" event={"ID":"2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3","Type":"ContainerStarted","Data":"eb8ec46b0b5420703f56f96aeec6fd3a12ff459a520a184b6455d60e38bd646a"} Jul 22 17:27:26.411606 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:26.411553 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g" podStartSLOduration=1.411538952 podStartE2EDuration="1.411538952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 17:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:27:26.410461625 +0000 UTC m=+2811.387189942" watchObservedRunningTime="2026-07-22 17:27:26.411538952 +0000 UTC m=+2811.388267270" Jul 22 17:27:27.400786 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:27.400750 2557 generic.go:356] "Generic (PLEG): container finished" podID="2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3" containerID="05192aea73954e684c246c8d7d69c1b48b28284d1ca64afe1e238293cee9f68d" exitCode=0 Jul 22 17:27:27.401194 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:27.400833 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g" event={"ID":"2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3","Type":"ContainerDied","Data":"05192aea73954e684c246c8d7d69c1b48b28284d1ca64afe1e238293cee9f68d"} Jul 22 17:27:28.541300 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:28.541273 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g" Jul 22 17:27:28.607698 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:28.607666 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3-maas-api-service-ca\") pod \"2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3\" (UID: \"2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3\") " Jul 22 17:27:28.608054 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:28.608001 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3" (UID: "2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:27:28.709222 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:28.709124 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3-maas-api-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:27:29.412102 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:29.412071 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g" Jul 22 17:27:29.412283 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:29.412070 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g" event={"ID":"2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3","Type":"ContainerDied","Data":"eb8ec46b0b5420703f56f96aeec6fd3a12ff459a520a184b6455d60e38bd646a"} Jul 22 17:27:29.412283 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:29.412218 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb8ec46b0b5420703f56f96aeec6fd3a12ff459a520a184b6455d60e38bd646a" Jul 22 17:27:35.816616 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:35.816581 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq"] Jul 22 17:27:35.817132 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:35.816780 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" podUID="3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5" containerName="maas-api" containerID="cri-o://e6da79149f7869eee2b89647168516e7b899aca3c9467f26a2b0eb9150acc13e" gracePeriod=30 Jul 22 17:27:35.846969 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:35.846946 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:27:36.070966 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.070909 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:27:36.174963 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.174930 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9rwx\" (UniqueName: \"kubernetes.io/projected/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-kube-api-access-m9rwx\") pod \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\" (UID: \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\") " Jul 22 17:27:36.175163 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.174997 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-maas-api-tls\") pod \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\" (UID: \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\") " Jul 22 17:27:36.175163 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.175080 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-openshift-service-ca\") pod \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\" (UID: \"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5\") " Jul 22 17:27:36.175483 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.175452 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5" (UID: "3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:27:36.177222 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.177199 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5" (UID: "3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:27:36.177321 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.177268 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-kube-api-access-m9rwx" (OuterVolumeSpecName: "kube-api-access-m9rwx") pod "3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5" (UID: "3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5"). InnerVolumeSpecName "kube-api-access-m9rwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:27:36.275923 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.275889 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-maas-api-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:27:36.275923 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.275920 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-openshift-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:27:36.275923 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.275930 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9rwx\" (UniqueName: \"kubernetes.io/projected/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5-kube-api-access-m9rwx\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:27:36.447447 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.447355 2557 generic.go:356] "Generic (PLEG): container finished" podID="3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5" containerID="e6da79149f7869eee2b89647168516e7b899aca3c9467f26a2b0eb9150acc13e" exitCode=0 Jul 22 17:27:36.447447 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.447400 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" Jul 22 17:27:36.447700 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.447441 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" event={"ID":"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5","Type":"ContainerDied","Data":"e6da79149f7869eee2b89647168516e7b899aca3c9467f26a2b0eb9150acc13e"} Jul 22 17:27:36.447700 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.447500 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq" event={"ID":"3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5","Type":"ContainerDied","Data":"918cdea168360c4ee02ce2d63d745948a1869f5f0ba78fd6554e44a98e312554"} Jul 22 17:27:36.447700 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.447529 2557 scope.go:117] "RemoveContainer" containerID="e6da79149f7869eee2b89647168516e7b899aca3c9467f26a2b0eb9150acc13e" Jul 22 17:27:36.462055 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.462035 2557 scope.go:117] "RemoveContainer" containerID="e6da79149f7869eee2b89647168516e7b899aca3c9467f26a2b0eb9150acc13e" Jul 22 17:27:36.462358 ip-10-0-132-94 kubenswrapper[2557]: E0722 17:27:36.462336 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6da79149f7869eee2b89647168516e7b899aca3c9467f26a2b0eb9150acc13e\": container with ID starting with e6da79149f7869eee2b89647168516e7b899aca3c9467f26a2b0eb9150acc13e not found: ID does not exist" containerID="e6da79149f7869eee2b89647168516e7b899aca3c9467f26a2b0eb9150acc13e" Jul 22 17:27:36.462404 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.462367 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6da79149f7869eee2b89647168516e7b899aca3c9467f26a2b0eb9150acc13e"} err="failed to get container status \"e6da79149f7869eee2b89647168516e7b899aca3c9467f26a2b0eb9150acc13e\": rpc error: code = NotFound desc = could not find container \"e6da79149f7869eee2b89647168516e7b899aca3c9467f26a2b0eb9150acc13e\": container with ID starting with e6da79149f7869eee2b89647168516e7b899aca3c9467f26a2b0eb9150acc13e not found: ID does not exist" Jul 22 17:27:36.472374 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.472346 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq"] Jul 22 17:27:36.476116 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:36.476092 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-629c13-79787c4cdc-pj6bq"] Jul 22 17:27:37.576692 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:37.576658 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5" path="/var/lib/kubelet/pods/3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5/volumes" Jul 22 17:27:40.522900 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:40.522867 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g"] Jul 22 17:27:40.526068 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:40.526044 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:27:40.526209 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:40.526145 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-629c13-95c4g"] Jul 22 17:27:40.914520 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:40.914444 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:27:41.574708 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:41.574676 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3" path="/var/lib/kubelet/pods/2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3/volumes" Jul 22 17:27:46.917901 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:46.917872 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:27:47.019198 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.019166 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:27:47.255984 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.255951 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:27:47.520781 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.520690 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:27:47.628494 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.628459 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc"] Jul 22 17:27:47.628943 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.628925 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5" containerName="maas-api" Jul 22 17:27:47.629062 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.628946 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5" containerName="maas-api" Jul 22 17:27:47.629062 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.628959 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3" containerName="revoke-keys" Jul 22 17:27:47.629062 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.628968 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3" containerName="revoke-keys" Jul 22 17:27:47.629225 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.629083 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb90ea1-f0d1-4ab0-bcad-5dd2a481c7a5" containerName="maas-api" Jul 22 17:27:47.629225 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.629098 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c0461fb-4bcc-4c9b-a8b7-aaaecd0502f3" containerName="revoke-keys" Jul 22 17:27:47.632193 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.632171 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc" Jul 22 17:27:47.635474 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.635455 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vln7b\"" Jul 22 17:27:47.641790 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.641763 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc"] Jul 22 17:27:47.788337 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.788250 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839a0531-9955-43df-bd9c-ab9d1a15108e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc\" (UID: \"839a0531-9955-43df-bd9c-ab9d1a15108e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc" Jul 22 17:27:47.888992 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.888961 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839a0531-9955-43df-bd9c-ab9d1a15108e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc\" (UID: \"839a0531-9955-43df-bd9c-ab9d1a15108e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc" Jul 22 17:27:47.889613 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.889593 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839a0531-9955-43df-bd9c-ab9d1a15108e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc\" (UID: \"839a0531-9955-43df-bd9c-ab9d1a15108e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc" Jul 22 17:27:47.945731 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:47.945697 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc" Jul 22 17:27:48.080183 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:48.080159 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc"] Jul 22 17:27:48.082410 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:27:48.082385 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839a0531_9955_43df_bd9c_ab9d1a15108e.slice/crio-767443ca09ccdf2aaf914d98a6da456192396f892abc3eef08750637af2abba1 WatchSource:0}: Error finding container 767443ca09ccdf2aaf914d98a6da456192396f892abc3eef08750637af2abba1: Status 404 returned error can't find the container with id 767443ca09ccdf2aaf914d98a6da456192396f892abc3eef08750637af2abba1 Jul 22 17:27:48.154893 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:48.154867 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:27:48.254253 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:48.254221 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:27:48.509619 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:48.509584 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc" event={"ID":"839a0531-9955-43df-bd9c-ab9d1a15108e","Type":"ContainerStarted","Data":"af47986af7ecd7c57cf48383bd6d4e6f9bff6786a3b34bbcf2c451afa2d0b403"} Jul 22 17:27:48.509776 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:48.509634 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc" event={"ID":"839a0531-9955-43df-bd9c-ab9d1a15108e","Type":"ContainerStarted","Data":"767443ca09ccdf2aaf914d98a6da456192396f892abc3eef08750637af2abba1"} Jul 22 17:27:49.521488 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:49.521451 2557 generic.go:356] "Generic (PLEG): container finished" podID="839a0531-9955-43df-bd9c-ab9d1a15108e" containerID="af47986af7ecd7c57cf48383bd6d4e6f9bff6786a3b34bbcf2c451afa2d0b403" exitCode=0 Jul 22 17:27:49.521889 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:49.521537 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc" event={"ID":"839a0531-9955-43df-bd9c-ab9d1a15108e","Type":"ContainerDied","Data":"af47986af7ecd7c57cf48383bd6d4e6f9bff6786a3b34bbcf2c451afa2d0b403"} Jul 22 17:27:50.662222 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:50.662194 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc" Jul 22 17:27:50.818207 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:50.818104 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839a0531-9955-43df-bd9c-ab9d1a15108e-maas-api-service-ca\") pod \"839a0531-9955-43df-bd9c-ab9d1a15108e\" (UID: \"839a0531-9955-43df-bd9c-ab9d1a15108e\") " Jul 22 17:27:50.818479 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:50.818458 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839a0531-9955-43df-bd9c-ab9d1a15108e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "839a0531-9955-43df-bd9c-ab9d1a15108e" (UID: "839a0531-9955-43df-bd9c-ab9d1a15108e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:27:50.919759 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:50.919726 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/839a0531-9955-43df-bd9c-ab9d1a15108e-maas-api-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:27:51.533421 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:51.533389 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc" event={"ID":"839a0531-9955-43df-bd9c-ab9d1a15108e","Type":"ContainerDied","Data":"767443ca09ccdf2aaf914d98a6da456192396f892abc3eef08750637af2abba1"} Jul 22 17:27:51.533421 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:51.533404 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc" Jul 22 17:27:51.533631 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:51.533435 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="767443ca09ccdf2aaf914d98a6da456192396f892abc3eef08750637af2abba1" Jul 22 17:27:58.329968 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.329926 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb"] Jul 22 17:27:58.330478 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.330197 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" podUID="37d986d8-ff16-44bd-a924-9180a88bc7e6" containerName="maas-api" containerID="cri-o://636f866c4ab82ba63685e0e3b99225abe8c5150c6881869ec6211d829b00b8bb" gracePeriod=30 Jul 22 17:27:58.340799 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.340775 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:27:58.573431 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.573399 2557 generic.go:356] "Generic (PLEG): container finished" podID="37d986d8-ff16-44bd-a924-9180a88bc7e6" containerID="636f866c4ab82ba63685e0e3b99225abe8c5150c6881869ec6211d829b00b8bb" exitCode=0 Jul 22 17:27:58.573567 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.573472 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" event={"ID":"37d986d8-ff16-44bd-a924-9180a88bc7e6","Type":"ContainerDied","Data":"636f866c4ab82ba63685e0e3b99225abe8c5150c6881869ec6211d829b00b8bb"} Jul 22 17:27:58.573567 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.573526 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" event={"ID":"37d986d8-ff16-44bd-a924-9180a88bc7e6","Type":"ContainerDied","Data":"309c1f00505dc51e930199e42eeecdf2eddf7c5e50df5680702b5b9e4aa4de4b"} Jul 22 17:27:58.573567 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.573544 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="309c1f00505dc51e930199e42eeecdf2eddf7c5e50df5680702b5b9e4aa4de4b" Jul 22 17:27:58.575576 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.575554 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:27:58.588180 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.588126 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7gn9\" (UniqueName: \"kubernetes.io/projected/37d986d8-ff16-44bd-a924-9180a88bc7e6-kube-api-access-c7gn9\") pod \"37d986d8-ff16-44bd-a924-9180a88bc7e6\" (UID: \"37d986d8-ff16-44bd-a924-9180a88bc7e6\") " Jul 22 17:27:58.588278 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.588183 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37d986d8-ff16-44bd-a924-9180a88bc7e6-maas-api-tls\") pod \"37d986d8-ff16-44bd-a924-9180a88bc7e6\" (UID: \"37d986d8-ff16-44bd-a924-9180a88bc7e6\") " Jul 22 17:27:58.588388 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.588373 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d986d8-ff16-44bd-a924-9180a88bc7e6-openshift-service-ca\") pod \"37d986d8-ff16-44bd-a924-9180a88bc7e6\" (UID: \"37d986d8-ff16-44bd-a924-9180a88bc7e6\") " Jul 22 17:27:58.588746 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.588716 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d986d8-ff16-44bd-a924-9180a88bc7e6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "37d986d8-ff16-44bd-a924-9180a88bc7e6" (UID: "37d986d8-ff16-44bd-a924-9180a88bc7e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 17:27:58.590239 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.590214 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d986d8-ff16-44bd-a924-9180a88bc7e6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "37d986d8-ff16-44bd-a924-9180a88bc7e6" (UID: "37d986d8-ff16-44bd-a924-9180a88bc7e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 17:27:58.590319 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.590251 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d986d8-ff16-44bd-a924-9180a88bc7e6-kube-api-access-c7gn9" (OuterVolumeSpecName: "kube-api-access-c7gn9") pod "37d986d8-ff16-44bd-a924-9180a88bc7e6" (UID: "37d986d8-ff16-44bd-a924-9180a88bc7e6"). InnerVolumeSpecName "kube-api-access-c7gn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 17:27:58.689448 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.689418 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7gn9\" (UniqueName: \"kubernetes.io/projected/37d986d8-ff16-44bd-a924-9180a88bc7e6-kube-api-access-c7gn9\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:27:58.689448 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.689444 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37d986d8-ff16-44bd-a924-9180a88bc7e6-maas-api-tls\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:27:58.689448 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:58.689455 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d986d8-ff16-44bd-a924-9180a88bc7e6-openshift-service-ca\") on node \"ip-10-0-132-94.ec2.internal\" DevicePath \"\"" Jul 22 17:27:59.577654 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:59.577624 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb" Jul 22 17:27:59.612522 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:59.612489 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb"] Jul 22 17:27:59.616475 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:27:59.616450 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-05e61d-8469ffbfcc-2r9vb"] Jul 22 17:28:01.580127 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:01.580088 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37d986d8-ff16-44bd-a924-9180a88bc7e6" path="/var/lib/kubelet/pods/37d986d8-ff16-44bd-a924-9180a88bc7e6/volumes" Jul 22 17:28:02.780401 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:02.780365 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:28:02.802403 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:02.802373 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc"] Jul 22 17:28:02.809016 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:02.808991 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-05e61d-6gzrc"] Jul 22 17:28:03.211259 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:03.211179 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:28:03.575541 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:03.575507 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="839a0531-9955-43df-bd9c-ab9d1a15108e" path="/var/lib/kubelet/pods/839a0531-9955-43df-bd9c-ab9d1a15108e/volumes" Jul 22 17:28:09.456774 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:09.456736 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:28:09.548257 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:09.548226 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:28:09.724943 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:09.724858 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:28:09.950208 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:09.950172 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:28:16.364366 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:16.364332 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58867cc484-6m6hv_e98623d2-9137-4308-bf23-93cba840f19a/manager/0.log" Jul 22 17:28:16.758113 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:16.758082 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55b7bf7664-8gr5j_e326eb82-b59a-4638-a7e2-e12b03bee5a0/copy-manifests/0.log" Jul 22 17:28:16.766697 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:16.766672 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55b7bf7664-8gr5j_e326eb82-b59a-4638-a7e2-e12b03bee5a0/manager/0.log" Jul 22 17:28:17.390285 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:17.390249 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-65996798d9-wb9dl_a6258786-d895-4d76-84da-b921ae0b611c/maas-api/0.log" Jul 22 17:28:17.543046 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:17.542996 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29745660-7w47l_61361050-4222-4246-8b76-678fc9638e58/cleanup/0.log" Jul 22 17:28:17.664052 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:17.663963 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29745675-pz6pc_6b085e8d-3a18-4ffe-ac43-1f21296d308f/cleanup/0.log" Jul 22 17:28:17.777697 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:17.777667 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-4w9t8_667d8c5d-35c7-48e5-8800-70b3f22aa3bf/postgres/0.log" Jul 22 17:28:20.465683 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:20.465655 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gch7l_1e71384d-5c47-4a2a-992c-c581ddf3669c/discovery/0.log" Jul 22 17:28:20.968276 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:20.968246 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-vg25t_02b632be-5f97-401e-abc1-036f2033c085/payload-pre-processing/0.log" Jul 22 17:28:21.083357 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:21.083327 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-fp786_eab14db0-c780-4ed8-9b19-93ee88b16537/payload-processing/0.log" Jul 22 17:28:21.565424 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:21.565395 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-p6p8l_162d131d-1c24-471b-8332-1e20c9e1b701/main/0.log" Jul 22 17:28:21.809616 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:21.809585 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-lz4qj_fea69359-b394-4b9c-8b44-fec16e5df47b/main/0.log" Jul 22 17:28:22.850097 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:22.850058 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:28:27.477341 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:27.477264 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:28:27.546312 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:27.546284 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:28:30.227828 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:30.227798 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4842j_27f20e9c-87e3-4473-ad01-99525c7b95e1/global-pull-secret-syncer/0.log" Jul 22 17:28:30.370469 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:30.370440 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" Jul 22 17:28:30.503704 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:30.503602 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-94.ec2.internal_466327f919d9276f8f0590ca18d52c6c/haproxy/0.log" Jul 22 17:28:35.966229 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:35.966138 2557 scope.go:117] "RemoveContainer" containerID="636f866c4ab82ba63685e0e3b99225abe8c5150c6881869ec6211d829b00b8bb" Jul 22 17:28:36.655149 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:36.655082 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5b4e4af1-8bab-400e-bdf0-d030a12220da/alertmanager/0.log" Jul 22 17:28:36.678690 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:36.678665 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5b4e4af1-8bab-400e-bdf0-d030a12220da/config-reloader/0.log" Jul 22 17:28:36.699141 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:36.699115 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5b4e4af1-8bab-400e-bdf0-d030a12220da/kube-rbac-proxy-web/0.log" Jul 22 17:28:36.722156 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:36.722134 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5b4e4af1-8bab-400e-bdf0-d030a12220da/kube-rbac-proxy/0.log" Jul 22 17:28:36.747931 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:36.747911 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5b4e4af1-8bab-400e-bdf0-d030a12220da/kube-rbac-proxy-metric/0.log" Jul 22 17:28:36.769549 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:36.769526 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5b4e4af1-8bab-400e-bdf0-d030a12220da/prom-label-proxy/0.log" Jul 22 17:28:36.789439 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:36.789418 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5b4e4af1-8bab-400e-bdf0-d030a12220da/init-config-reloader/0.log" Jul 22 17:28:36.865229 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:36.865204 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-mpf4c_a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db/kube-state-metrics/0.log" Jul 22 17:28:36.881755 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:36.881732 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-mpf4c_a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db/kube-rbac-proxy-main/0.log" Jul 22 17:28:36.901230 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:36.901204 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-mpf4c_a5ed1995-b3a8-4c3c-b0a5-3ed1a1a147db/kube-rbac-proxy-self/0.log" Jul 22 17:28:36.926226 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:36.926160 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-77c5f697f5-9ckm9_ade6b05e-41d7-43fe-b7e3-0bdc444fad5e/metrics-server/0.log" Jul 22 17:28:36.963426 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:36.963400 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-2ndfc_a5d6a867-9a45-4387-983a-99649541632b/monitoring-plugin/0.log" Jul 22 17:28:37.035391 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.035366 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lqf5c_d321f9a7-ee1f-4c09-aab2-157e849f71f3/node-exporter/0.log" Jul 22 17:28:37.092548 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.092526 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lqf5c_d321f9a7-ee1f-4c09-aab2-157e849f71f3/kube-rbac-proxy/0.log" Jul 22 17:28:37.128598 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.128572 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lqf5c_d321f9a7-ee1f-4c09-aab2-157e849f71f3/init-textfile/0.log" Jul 22 17:28:37.327155 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.327127 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-wz62t_12249663-3e2d-425c-a1ed-76788ad82cbd/kube-rbac-proxy-main/0.log" Jul 22 17:28:37.352265 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.352242 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-wz62t_12249663-3e2d-425c-a1ed-76788ad82cbd/kube-rbac-proxy-self/0.log" Jul 22 17:28:37.372585 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.372565 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-wz62t_12249663-3e2d-425c-a1ed-76788ad82cbd/openshift-state-metrics/0.log" Jul 22 17:28:37.419508 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.419483 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0eb3d5b7-040c-4232-bd15-54c61a468c0f/prometheus/0.log" Jul 22 17:28:37.444091 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.444069 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0eb3d5b7-040c-4232-bd15-54c61a468c0f/config-reloader/0.log" Jul 22 17:28:37.469579 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.469556 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0eb3d5b7-040c-4232-bd15-54c61a468c0f/thanos-sidecar/0.log" Jul 22 17:28:37.497112 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.497088 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0eb3d5b7-040c-4232-bd15-54c61a468c0f/kube-rbac-proxy-web/0.log" Jul 22 17:28:37.536908 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.536887 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0eb3d5b7-040c-4232-bd15-54c61a468c0f/kube-rbac-proxy/0.log" Jul 22 17:28:37.557104 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.557079 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0eb3d5b7-040c-4232-bd15-54c61a468c0f/kube-rbac-proxy-thanos/0.log" Jul 22 17:28:37.586640 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.586619 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0eb3d5b7-040c-4232-bd15-54c61a468c0f/init-config-reloader/0.log" Jul 22 17:28:37.630589 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.630567 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-pfnpc_4487fd14-d796-4638-8f4d-731fad568b10/prometheus-operator/0.log" Jul 22 17:28:37.651315 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.651273 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-pfnpc_4487fd14-d796-4638-8f4d-731fad568b10/kube-rbac-proxy/0.log" Jul 22 17:28:37.789164 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.789136 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7675958d48-7ndcw_f35a1044-4f56-4b49-b1f8-73d70c28b7b3/thanos-query/0.log" Jul 22 17:28:37.819863 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.819836 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7675958d48-7ndcw_f35a1044-4f56-4b49-b1f8-73d70c28b7b3/kube-rbac-proxy-web/0.log" Jul 22 17:28:37.845149 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.845091 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7675958d48-7ndcw_f35a1044-4f56-4b49-b1f8-73d70c28b7b3/kube-rbac-proxy/0.log" Jul 22 17:28:37.864578 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.864557 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7675958d48-7ndcw_f35a1044-4f56-4b49-b1f8-73d70c28b7b3/prom-label-proxy/0.log" Jul 22 17:28:37.885994 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.885980 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7675958d48-7ndcw_f35a1044-4f56-4b49-b1f8-73d70c28b7b3/kube-rbac-proxy-rules/0.log" Jul 22 17:28:37.909343 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:37.909326 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7675958d48-7ndcw_f35a1044-4f56-4b49-b1f8-73d70c28b7b3/kube-rbac-proxy-metrics/0.log" Jul 22 17:28:40.660609 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.660571 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw"] Jul 22 17:28:40.661208 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.660938 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d986d8-ff16-44bd-a924-9180a88bc7e6" containerName="maas-api" Jul 22 17:28:40.661208 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.660949 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d986d8-ff16-44bd-a924-9180a88bc7e6" containerName="maas-api" Jul 22 17:28:40.661208 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.660975 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="839a0531-9955-43df-bd9c-ab9d1a15108e" containerName="revoke-keys" Jul 22 17:28:40.661208 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.660981 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="839a0531-9955-43df-bd9c-ab9d1a15108e" containerName="revoke-keys" Jul 22 17:28:40.661208 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.661064 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="839a0531-9955-43df-bd9c-ab9d1a15108e" containerName="revoke-keys" Jul 22 17:28:40.661208 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.661076 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d986d8-ff16-44bd-a924-9180a88bc7e6" containerName="maas-api" Jul 22 17:28:40.666006 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.665986 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.668714 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.668695 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-52llv\"/\"kube-root-ca.crt\"" Jul 22 17:28:40.669980 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.669815 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-52llv\"/\"openshift-service-ca.crt\"" Jul 22 17:28:40.670207 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.670187 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-52llv\"/\"default-dockercfg-rh2ng\"" Jul 22 17:28:40.672466 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.672446 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw"] Jul 22 17:28:40.678092 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.678069 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5641cb62-90fb-462a-8147-7678d1b901e6-lib-modules\") pod \"perf-node-gather-daemonset-nw6tw\" (UID: \"5641cb62-90fb-462a-8147-7678d1b901e6\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.678206 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.678098 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5641cb62-90fb-462a-8147-7678d1b901e6-podres\") pod \"perf-node-gather-daemonset-nw6tw\" (UID: \"5641cb62-90fb-462a-8147-7678d1b901e6\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.678206 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.678126 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5641cb62-90fb-462a-8147-7678d1b901e6-sys\") pod \"perf-node-gather-daemonset-nw6tw\" (UID: \"5641cb62-90fb-462a-8147-7678d1b901e6\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.678206 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.678178 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5641cb62-90fb-462a-8147-7678d1b901e6-proc\") pod \"perf-node-gather-daemonset-nw6tw\" (UID: \"5641cb62-90fb-462a-8147-7678d1b901e6\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.678389 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.678222 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwjls\" (UniqueName: \"kubernetes.io/projected/5641cb62-90fb-462a-8147-7678d1b901e6-kube-api-access-vwjls\") pod \"perf-node-gather-daemonset-nw6tw\" (UID: \"5641cb62-90fb-462a-8147-7678d1b901e6\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.699633 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.699611 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-phdn9_19ea0909-e6e0-419a-a6cd-31bcb406be6e/download-server/0.log" Jul 22 17:28:40.779328 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.779295 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5641cb62-90fb-462a-8147-7678d1b901e6-sys\") pod \"perf-node-gather-daemonset-nw6tw\" (UID: \"5641cb62-90fb-462a-8147-7678d1b901e6\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.779513 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.779342 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5641cb62-90fb-462a-8147-7678d1b901e6-proc\") pod \"perf-node-gather-daemonset-nw6tw\" (UID: \"5641cb62-90fb-462a-8147-7678d1b901e6\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.779513 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.779381 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwjls\" (UniqueName: \"kubernetes.io/projected/5641cb62-90fb-462a-8147-7678d1b901e6-kube-api-access-vwjls\") pod \"perf-node-gather-daemonset-nw6tw\" (UID: \"5641cb62-90fb-462a-8147-7678d1b901e6\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.779513 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.779414 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5641cb62-90fb-462a-8147-7678d1b901e6-sys\") pod \"perf-node-gather-daemonset-nw6tw\" (UID: \"5641cb62-90fb-462a-8147-7678d1b901e6\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.779513 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.779436 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5641cb62-90fb-462a-8147-7678d1b901e6-lib-modules\") pod \"perf-node-gather-daemonset-nw6tw\" (UID: \"5641cb62-90fb-462a-8147-7678d1b901e6\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.779513 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.779453 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5641cb62-90fb-462a-8147-7678d1b901e6-podres\") pod \"perf-node-gather-daemonset-nw6tw\" (UID: \"5641cb62-90fb-462a-8147-7678d1b901e6\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.779513 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.779494 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5641cb62-90fb-462a-8147-7678d1b901e6-proc\") pod \"perf-node-gather-daemonset-nw6tw\" (UID: \"5641cb62-90fb-462a-8147-7678d1b901e6\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.779732 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.779536 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5641cb62-90fb-462a-8147-7678d1b901e6-lib-modules\") pod \"perf-node-gather-daemonset-nw6tw\" (UID: \"5641cb62-90fb-462a-8147-7678d1b901e6\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.779732 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.779555 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5641cb62-90fb-462a-8147-7678d1b901e6-podres\") pod \"perf-node-gather-daemonset-nw6tw\" (UID: \"5641cb62-90fb-462a-8147-7678d1b901e6\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.788590 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.788563 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwjls\" (UniqueName: \"kubernetes.io/projected/5641cb62-90fb-462a-8147-7678d1b901e6-kube-api-access-vwjls\") pod \"perf-node-gather-daemonset-nw6tw\" (UID: \"5641cb62-90fb-462a-8147-7678d1b901e6\") " pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:40.976513 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:40.976491 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:41.104214 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:41.104190 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw"] Jul 22 17:28:41.106323 ip-10-0-132-94 kubenswrapper[2557]: W0722 17:28:41.106296 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5641cb62_90fb_462a_8147_7678d1b901e6.slice/crio-d0293546a31db465c9319a35ae9c35c1b075fc30ee27417c224e6e204bb3ec63 WatchSource:0}: Error finding container d0293546a31db465c9319a35ae9c35c1b075fc30ee27417c224e6e204bb3ec63: Status 404 returned error can't find the container with id d0293546a31db465c9319a35ae9c35c1b075fc30ee27417c224e6e204bb3ec63 Jul 22 17:28:41.795668 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:41.795634 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" event={"ID":"5641cb62-90fb-462a-8147-7678d1b901e6","Type":"ContainerStarted","Data":"9b4d7e9d8bc14a81f98db085857a69df873ee64b1df05a904afa4873b62dfbf5"} Jul 22 17:28:41.796183 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:41.795695 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:41.796183 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:41.795715 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" event={"ID":"5641cb62-90fb-462a-8147-7678d1b901e6","Type":"ContainerStarted","Data":"d0293546a31db465c9319a35ae9c35c1b075fc30ee27417c224e6e204bb3ec63"} Jul 22 17:28:41.812827 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:41.812778 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" podStartSLOduration=1.8127633140000001 podStartE2EDuration="1.812763314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 17:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 17:28:41.81080193 +0000 UTC m=+2886.787530246" watchObservedRunningTime="2026-07-22 17:28:41.812763314 +0000 UTC m=+2886.789491629" Jul 22 17:28:42.422641 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:42.422612 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gbnnc_f4c6bfe9-79f1-4d51-bc5f-04d11054953e/dns/0.log" Jul 22 17:28:42.443402 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:42.443378 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gbnnc_f4c6bfe9-79f1-4d51-bc5f-04d11054953e/kube-rbac-proxy/0.log" Jul 22 17:28:42.579695 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:42.579666 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f7wc2_6acdc380-0278-49e9-a977-f16754ea4d48/dns-node-resolver/0.log" Jul 22 17:28:43.162066 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:43.161042 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hfg9n_f0c4cd2b-33ca-47fe-8263-dab2a28569cc/node-ca/0.log" Jul 22 17:28:44.193147 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:44.193117 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gch7l_1e71384d-5c47-4a2a-992c-c581ddf3669c/discovery/0.log" Jul 22 17:28:44.371726 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:44.371696 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-vg25t_02b632be-5f97-401e-abc1-036f2033c085/payload-pre-processing/0.log" Jul 22 17:28:44.400350 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:44.400326 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-fp786_eab14db0-c780-4ed8-9b19-93ee88b16537/payload-processing/0.log" Jul 22 17:28:45.115582 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:45.115551 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g5wlp_f58b7e81-5b73-4f25-8194-1869ad43e260/serve-healthcheck-canary/0.log" Jul 22 17:28:45.684298 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:45.684266 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d9zkt_2e46da55-c83c-410b-95d5-e6d5af057a64/kube-rbac-proxy/0.log" Jul 22 17:28:45.702873 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:45.702848 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d9zkt_2e46da55-c83c-410b-95d5-e6d5af057a64/exporter/0.log" Jul 22 17:28:45.722135 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:45.722107 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d9zkt_2e46da55-c83c-410b-95d5-e6d5af057a64/extractor/0.log" Jul 22 17:28:47.811867 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:47.811831 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-52llv/perf-node-gather-daemonset-nw6tw" Jul 22 17:28:48.342809 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:48.342777 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58867cc484-6m6hv_e98623d2-9137-4308-bf23-93cba840f19a/manager/0.log" Jul 22 17:28:48.453733 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:48.453701 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55b7bf7664-8gr5j_e326eb82-b59a-4638-a7e2-e12b03bee5a0/manager/0.log" Jul 22 17:28:48.473435 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:48.473409 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55b7bf7664-8gr5j_e326eb82-b59a-4638-a7e2-e12b03bee5a0/copy-manifests/0.log" Jul 22 17:28:49.290215 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:49.290187 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lmvp_c831c7ba-6c28-4355-a3e1-271edff8c1ad/konnectivity-agent/0.log" limit=1024 Jul 22 17:28:50.075175 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:50.075098 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57844c55d9-dc7mq_783ac916-7082-4f82-a3bd-a0a1630be59e/manager/0.log" Jul 22 17:28:50.098980 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:50.098954 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ddlg5_b3d6f982-55c0-45a0-8ab6-8fdad5d2873c/openshift-lws-operator/0.log" Jul 22 17:28:50.808813 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:50.808721 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gch7l_1e71384d-5c47-4a2a-992c-c581ddf3669c/discovery/0.log" Jul 22 17:28:50.944765 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:50.944737 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-vg25t_02b632be-5f97-401e-abc1-036f2033c085/payload-pre-processing/0.log" Jul 22 17:28:50.965504 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:50.965478 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-fp786_eab14db0-c780-4ed8-9b19-93ee88b16537/payload-processing/0.log" Jul 22 17:28:52.234662 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:52.234632 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gch7l_1e71384d-5c47-4a2a-992c-c581ddf3669c/discovery/0.log" Jul 22 17:28:52.365083 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:52.364991 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-vg25t_02b632be-5f97-401e-abc1-036f2033c085/payload-pre-processing/0.log" Jul 22 17:28:52.394771 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:52.394745 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-fp786_eab14db0-c780-4ed8-9b19-93ee88b16537/payload-processing/0.log" Jul 22 17:28:56.687214 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:56.687187 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncfqd_ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6/kube-multus-additional-cni-plugins/0.log" Jul 22 17:28:56.706136 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:56.706112 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncfqd_ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6/egress-router-binary-copy/0.log" Jul 22 17:28:56.724292 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:56.724268 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncfqd_ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6/cni-plugins/0.log" Jul 22 17:28:56.744169 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:56.744143 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncfqd_ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6/bond-cni-plugin/0.log" Jul 22 17:28:56.763307 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:56.763279 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncfqd_ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6/routeoverride-cni/0.log" Jul 22 17:28:56.781991 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:56.781961 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncfqd_ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6/whereabouts-cni-bincopy/0.log" Jul 22 17:28:56.801141 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:56.801116 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncfqd_ca2a4f4f-1df3-4d41-b2f7-4065936ff6f6/whereabouts-cni/0.log" Jul 22 17:28:57.031966 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:57.031934 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qb6fj_f202c1ae-315f-4f19-b0a3-706a278e1177/kube-multus/0.log" Jul 22 17:28:57.082117 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:57.082084 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-65bkg_9704c18f-5f2d-417b-b49f-bf27e2774eef/network-metrics-daemon/0.log" Jul 22 17:28:57.098307 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:57.098277 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-65bkg_9704c18f-5f2d-417b-b49f-bf27e2774eef/kube-rbac-proxy/0.log" Jul 22 17:28:58.044188 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:58.044158 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bs4s_362a0176-19e5-42d8-808c-e32e578f7527/ovn-controller/0.log" Jul 22 17:28:58.064086 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:58.064056 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bs4s_362a0176-19e5-42d8-808c-e32e578f7527/ovn-acl-logging/0.log" Jul 22 17:28:58.090787 ip-10-0-132-94 kubenswrapper[2557]: I0722 17:28:58.090762 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bs4s_362a0176-19e5-42d8-808c-e32e578f7527/kube-rbac-proxy-node/0.log"