Jul 15 11:58:52.457044 ip-10-0-131-53 systemd[1]: Starting Kubernetes Kubelet... Jul 15 11:58:52.908758 ip-10-0-131-53 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 11:58:52.908758 ip-10-0-131-53 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 11:58:52.908758 ip-10-0-131-53 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 11:58:52.909434 ip-10-0-131-53 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 11:58:52.909434 ip-10-0-131-53 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 11:58:52.910980 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.910851 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 11:58:52.918502 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918478 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 11:58:52.918502 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918500 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 11:58:52.918502 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918504 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918507 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918511 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918514 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918518 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918521 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918523 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918526 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918529 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918531 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918534 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918537 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918539 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918548 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918551 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918553 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918556 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918565 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918568 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918571 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 11:58:52.918608 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918574 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918577 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918579 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918582 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918584 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918586 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918589 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918591 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918594 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918596 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918599 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918603 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918607 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918610 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918613 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918618 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918622 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918625 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 11:58:52.919106 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918629 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918633 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918636 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918639 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918642 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918645 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918648 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918651 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918656 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918658 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918661 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918664 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918666 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918669 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918671 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918673 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918680 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918683 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918686 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 11:58:52.919541 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918688 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918691 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918694 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918696 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918699 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918702 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918705 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918707 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918711 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918713 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918716 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918719 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918721 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918724 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918726 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918729 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918731 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918733 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918758 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918761 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 11:58:52.920062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918764 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918767 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918770 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918773 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918775 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918779 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918781 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918783 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918786 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918790 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918793 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918795 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918798 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918800 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918803 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918805 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918808 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918810 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918813 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.918815 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 11:58:52.920544 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919546 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919553 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919555 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919558 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919561 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919564 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919566 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919568 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919571 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919574 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919577 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919579 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919582 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919584 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919587 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919590 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919592 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919595 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919597 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919600 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 11:58:52.921039 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919602 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919607 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919609 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919612 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919614 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919617 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919619 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919621 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919624 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919627 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919629 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919631 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919634 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919637 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919640 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919643 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919645 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919648 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919650 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919653 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 11:58:52.921542 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919655 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919657 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919660 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919662 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919665 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919667 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919669 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919673 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919678 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919681 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919684 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919686 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919689 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919693 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919696 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919698 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919701 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919703 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919706 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919708 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 11:58:52.922042 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919710 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919713 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919717 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919721 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919724 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919726 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919729 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919732 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919755 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919758 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919760 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919763 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919765 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919768 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919770 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919773 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919775 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919778 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919780 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 11:58:52.922532 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919784 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919786 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919789 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919792 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919794 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919797 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919801 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919804 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919806 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919809 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919811 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919814 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919818 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919821 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919825 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919827 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919831 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919833 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919836 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 11:58:52.923111 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.919838 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920242 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920247 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920250 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920252 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920255 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920257 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920260 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920262 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920265 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920267 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920270 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920273 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920275 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920277 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920280 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920283 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920285 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920288 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920292 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 11:58:52.923862 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920294 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920297 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920299 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920301 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920304 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920306 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920309 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920313 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920317 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920320 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920322 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920325 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920327 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920330 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920333 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920336 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920338 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920342 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920344 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 11:58:52.924395 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920346 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920349 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920351 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920353 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920356 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920358 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920361 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920363 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920366 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920368 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920371 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920373 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920377 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920379 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920382 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920384 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920386 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920389 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920391 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920393 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 11:58:52.924875 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920396 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920398 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920401 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920403 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920406 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920408 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920411 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920413 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920416 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920418 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920420 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920423 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920425 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920427 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920430 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920432 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920434 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920438 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920440 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 11:58:52.925390 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920444 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920447 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920449 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920452 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920455 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920458 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920461 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920463 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920466 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920468 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920470 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920474 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920478 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920480 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920483 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920486 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920489 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920492 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920494 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 11:58:52.925867 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920497 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920500 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.920502 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920584 2570 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920604 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920613 2570 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920620 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920626 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920630 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920634 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920638 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920642 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920644 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920648 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920652 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920655 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920658 2570 flags.go:64] FLAG: --cgroup-root="" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920661 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920666 2570 flags.go:64] FLAG: --client-ca-file="" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920669 2570 flags.go:64] FLAG: --cloud-provider="external" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920672 2570 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920676 2570 flags.go:64] FLAG: --cluster-domain="" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920679 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920682 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 11:58:52.926325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920685 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920689 2570 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920692 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920698 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920701 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920705 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920707 2570 flags.go:64] FLAG: --contention-profiling="false" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920710 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920713 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920716 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920719 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920723 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920726 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920729 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920732 2570 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920747 2570 flags.go:64] FLAG: --enable-server="true" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920751 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920755 2570 flags.go:64] FLAG: --event-burst="100" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920758 2570 flags.go:64] FLAG: --event-qps="50" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920761 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920764 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920767 2570 flags.go:64] FLAG: --eviction-hard="" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920770 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920773 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920776 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 11:58:52.927022 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920784 2570 flags.go:64] FLAG: --eviction-soft="" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920789 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920792 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920795 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920798 2570 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920801 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920804 2570 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920806 2570 flags.go:64] FLAG: --feature-gates="" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920810 2570 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920813 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920817 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920821 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920824 2570 flags.go:64] FLAG: --healthz-port="10248" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920827 2570 flags.go:64] FLAG: --help="false" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920830 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-131-53.ec2.internal" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920833 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920835 2570 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920838 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920841 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920845 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920847 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920850 2570 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920853 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 11:58:52.927621 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920856 2570 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920858 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920861 2570 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920864 2570 flags.go:64] FLAG: --kube-reserved="" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920867 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920870 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920872 2570 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920875 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920878 2570 flags.go:64] FLAG: --lock-file="" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920881 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920886 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920889 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920895 2570 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920898 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920901 2570 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920904 2570 flags.go:64] FLAG: --logging-format="text" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920907 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920910 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920913 2570 flags.go:64] FLAG: --manifest-url="" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920917 2570 flags.go:64] FLAG: --manifest-url-header="" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920922 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920925 2570 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920929 2570 flags.go:64] FLAG: --max-pods="110" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920932 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920935 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 11:58:52.928238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920938 2570 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920941 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920944 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920947 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920950 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920958 2570 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920961 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920964 2570 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920967 2570 flags.go:64] FLAG: --pod-cidr="" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920970 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920975 2570 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920978 2570 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920981 2570 flags.go:64] FLAG: --pods-per-core="0" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920984 2570 flags.go:64] FLAG: --port="10250" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920987 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920990 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01f0a0c19edab6c9e" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920993 2570 flags.go:64] FLAG: --qos-reserved="" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.920995 2570 flags.go:64] FLAG: --read-only-port="10255" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921000 2570 flags.go:64] FLAG: --register-node="true" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921003 2570 flags.go:64] FLAG: --register-with-taints="" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921007 2570 flags.go:64] FLAG: --registry-burst="10" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921010 2570 flags.go:64] FLAG: --registry-qps="5" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921013 2570 flags.go:64] FLAG: --reserved-cpus="" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921015 2570 flags.go:64] FLAG: --reserved-memory="" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921019 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 11:58:52.928916 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921022 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921026 2570 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921028 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921031 2570 flags.go:64] FLAG: --runonce="false" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921034 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921037 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921040 2570 flags.go:64] FLAG: --seccomp-default="false" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921043 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921047 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921050 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921053 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921056 2570 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921059 2570 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921062 2570 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921065 2570 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921067 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921070 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921073 2570 flags.go:64] FLAG: --system-cgroups="" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921076 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921081 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921084 2570 flags.go:64] FLAG: --tls-cert-file="" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921087 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921090 2570 flags.go:64] FLAG: --tls-min-version="" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921093 2570 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921095 2570 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 11:58:52.929558 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921099 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921102 2570 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921105 2570 flags.go:64] FLAG: --v="2" Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921109 2570 flags.go:64] FLAG: --version="false" Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921113 2570 flags.go:64] FLAG: --vmodule="" Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921117 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.921120 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921591 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921599 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921602 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921605 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921607 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921610 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921612 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921614 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921618 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921622 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921624 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921627 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921629 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921632 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 11:58:52.930184 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921634 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921637 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921639 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921641 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921644 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921646 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921648 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921651 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921653 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921656 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921658 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921661 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921664 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921666 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921669 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921672 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921674 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921677 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921680 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 11:58:52.930696 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921683 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921686 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921689 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921691 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921694 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921696 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921698 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921701 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921703 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921705 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921709 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921712 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921715 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921718 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921721 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921724 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921726 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921729 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921731 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 11:58:52.931189 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921747 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921749 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921752 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921754 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921757 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921759 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921762 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921764 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921767 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921770 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921773 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921775 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921778 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921782 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921784 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921787 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921791 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921794 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921797 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 11:58:52.931652 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921799 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921802 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921805 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921807 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921810 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921812 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921814 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921817 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921819 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921822 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921824 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921827 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921829 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921832 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921834 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921837 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921839 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921841 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921844 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921846 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 11:58:52.932137 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921849 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 11:58:52.932623 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921851 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 11:58:52.932623 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921854 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 11:58:52.932623 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921856 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 11:58:52.932623 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921859 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 11:58:52.932623 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921861 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 11:58:52.932623 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921865 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 11:58:52.932623 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.921868 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 11:58:52.932623 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.922731 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 11:58:52.932623 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.930644 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 11:58:52.932623 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.930663 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 11:58:52.932623 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.931585 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 11:58:52.932623 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.931594 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 11:58:52.932623 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931651 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 11:58:52.932623 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931658 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 11:58:52.932623 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931662 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931666 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931669 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931672 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931675 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931678 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931680 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931683 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931685 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931688 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931690 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931693 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931696 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931699 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931701 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931704 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931706 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931709 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931711 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931713 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 11:58:52.933015 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931716 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931718 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931721 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931723 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931726 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931728 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931731 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931750 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931753 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931756 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931759 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931761 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931764 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931767 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931769 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931772 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931775 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931777 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931780 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931782 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 11:58:52.933501 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931785 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931788 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931790 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931793 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931795 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931798 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931800 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931803 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931805 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931807 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931810 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931812 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931815 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931818 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931820 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931823 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931826 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931828 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931831 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931834 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 11:58:52.934005 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931836 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931839 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931841 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931844 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931846 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931849 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931851 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931854 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931856 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931859 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931862 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931864 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931868 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931872 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931875 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931879 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931883 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931885 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931889 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 11:58:52.934509 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931892 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931894 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931897 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931900 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931903 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931905 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931908 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931910 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931914 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931916 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931919 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931922 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931924 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931927 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931930 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931934 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931938 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 11:58:52.934995 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.931941 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 11:58:52.935408 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.931947 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 11:58:52.935408 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932068 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 11:58:52.935408 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932073 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 11:58:52.935408 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932076 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 11:58:52.935408 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932080 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 11:58:52.935408 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932083 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 11:58:52.935408 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932085 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 11:58:52.935408 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932089 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 11:58:52.935408 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932091 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 11:58:52.935408 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932094 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 11:58:52.935408 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932096 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 11:58:52.935408 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932099 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 11:58:52.935408 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932102 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 11:58:52.935408 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932105 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 11:58:52.935408 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932108 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932110 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932113 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932115 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932118 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932120 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932122 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932124 2570 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932127 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932130 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932133 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932135 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932138 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932140 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932143 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932145 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932148 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932150 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932153 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932156 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 11:58:52.935842 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932158 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932160 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932163 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932166 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932169 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932172 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932174 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932177 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932179 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932183 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932186 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932190 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932192 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932195 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932199 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932202 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932204 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932207 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932210 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 11:58:52.936353 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932212 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932215 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932217 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932220 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932222 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932225 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932227 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932230 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932232 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932235 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932237 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932240 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932242 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932246 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932249 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932252 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932255 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932257 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932259 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 11:58:52.936843 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932262 2570 feature_gate.go:328] unrecognized feature gate: Example Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932264 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932267 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932269 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932273 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932276 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932279 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932282 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932284 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932287 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932290 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932292 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932294 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932297 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932299 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932301 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932304 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932306 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932309 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932311 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 11:58:52.937291 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932314 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 11:58:52.937817 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932316 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 11:58:52.937817 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932318 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 11:58:52.937817 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932321 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 11:58:52.937817 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932323 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 11:58:52.937817 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932326 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 11:58:52.937817 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932328 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 11:58:52.937817 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:52.932331 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 11:58:52.937817 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.932336 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 11:58:52.937817 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.932493 2570 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 11:58:52.937817 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.936075 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 11:58:52.937817 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.937659 2570 server.go:1019] "Starting client certificate rotation" Jul 15 11:58:52.937817 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.937783 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 11:58:52.938366 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.937834 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 11:58:52.940725 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.940709 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 11:58:52.943004 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.942985 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 11:58:52.957697 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.957671 2570 log.go:25] "Validated CRI v1 runtime API" Jul 15 11:58:52.960229 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.960213 2570 log.go:25] "Validated CRI v1 image API" Jul 15 11:58:52.961053 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.961033 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 11:58:52.967180 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.967157 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 11:58:52.967415 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.967386 2570 fs.go:135] Filesystem UUIDs: map[5e463aee-fadc-4bae-8a50-0ab64577b08f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7b592c96-279b-42da-b609-a127417435c2:/dev/nvme0n1p4] Jul 15 11:58:52.967500 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.967414 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 11:58:52.972608 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.972495 2570 manager.go:216] Machine: {Timestamp:2026-07-15 11:58:52.971145786 +0000 UTC m=+0.384674129 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100001 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f671b84c26c196b8cfe5b88d8d580 SystemUUID:ec2f671b-84c2-6c19-6b8c-fe5b88d8d580 BootID:9cfa12b5-00e6-412a-bb37-d272b720a372 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:19:49:00:cd:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:19:49:00:cd:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:5e:9a:e4:4a:01 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 11:58:52.972608 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.972601 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 11:58:52.972759 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.972686 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 11:58:52.974717 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.974687 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 11:58:52.974878 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.974719 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-53.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 11:58:52.975579 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.975569 2570 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 11:58:52.975612 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.975581 2570 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 11:58:52.975612 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.975590 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 11:58:52.975612 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.975605 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 11:58:52.977652 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.977641 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 11:58:52.981424 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.981411 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 15 11:58:52.981544 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.981535 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 11:58:52.981582 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.981566 2570 kubelet.go:486] "Attempting to sync node with API server" Jul 15 11:58:52.981582 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.981577 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 11:58:52.981647 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.981590 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 11:58:52.981647 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.981600 2570 kubelet.go:398] "Adding apiserver pod source" Jul 15 11:58:52.981647 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.981612 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 11:58:52.982005 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.981991 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 11:58:52.982082 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.982026 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 11:58:52.985403 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.985384 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 11:58:52.987778 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.987761 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 11:58:52.987860 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.987800 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 11:58:52.988055 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.988041 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 11:58:52.988113 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.988061 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 11:58:52.988113 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.988070 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 11:58:52.988113 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.988078 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 11:58:52.988113 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.988091 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 11:58:52.988113 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.988100 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 11:58:52.988619 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.988607 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 11:58:52.988667 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.988622 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 11:58:52.988667 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.988643 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 11:58:52.988667 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.988653 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 11:58:52.988805 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.988672 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 11:58:52.988805 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.988685 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 11:58:52.990498 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.990485 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 11:58:52.990498 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.990500 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 11:58:52.992806 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.992774 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m56qn" Jul 15 11:58:52.993915 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.993902 2570 server.go:1268] "Started kubelet" Jul 15 11:58:52.994034 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.994002 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 11:58:52.994127 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.994067 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 11:58:52.994187 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.994160 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 11:58:52.994845 ip-10-0-131-53 systemd[1]: Started Kubernetes Kubelet. Jul 15 11:58:52.995227 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.995204 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-53.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 11:58:52.995824 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.995714 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 11:58:52.996049 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:52.996018 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 11:58:52.996121 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:52.996020 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-53.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 11:58:52.996508 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:52.996497 2570 server.go:310] "Adding debug handlers to kubelet server" Jul 15 11:58:53.001788 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.001759 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 11:58:53.002811 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.002790 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m56qn" Jul 15 11:58:53.009371 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.008200 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-53.ec2.internal.18c274b74bee1d62 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-53.ec2.internal,UID:ip-10-0-131-53.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-53.ec2.internal,},FirstTimestamp:2026-07-15 11:58:52.993871202 +0000 UTC m=+0.407399544,LastTimestamp:2026-07-15 11:58:52.993871202 +0000 UTC m=+0.407399544,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-53.ec2.internal,}" Jul 15 11:58:53.009463 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.009387 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 11:58:53.009463 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.009395 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 11:58:53.009706 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.009685 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:53.010243 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.010222 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 11:58:53.010243 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.010227 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 11:58:53.010404 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.010388 2570 factory.go:55] Registering systemd factory Jul 15 11:58:53.010451 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.010407 2570 factory.go:223] Registration of the systemd container factory successfully Jul 15 11:58:53.010531 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.010521 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 11:58:53.010649 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.010636 2570 reconstruct.go:97] "Volume reconstruction finished" Jul 15 11:58:53.010693 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.010650 2570 reconciler.go:29] "Reconciler: start to sync state" Jul 15 11:58:53.010823 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.010810 2570 factory.go:153] Registering CRI-O factory Jul 15 11:58:53.010899 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.010827 2570 factory.go:223] Registration of the crio container factory successfully Jul 15 11:58:53.010899 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.010877 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 11:58:53.010899 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.010898 2570 factory.go:103] Registering Raw factory Jul 15 11:58:53.011203 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.010915 2570 manager.go:1196] Started watching for new ooms in manager Jul 15 11:58:53.011557 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.011535 2570 manager.go:349] Starting recovery of all containers Jul 15 11:58:53.021135 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.021108 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 11:58:53.022806 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.022791 2570 manager.go:354] Recovery completed Jul 15 11:58:53.024600 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.024579 2570 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-53.ec2.internal\" not found" node="ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.027174 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.027158 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 11:58:53.030253 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.030229 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasSufficientMemory" Jul 15 11:58:53.030343 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.030269 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 11:58:53.030343 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.030289 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasSufficientPID" Jul 15 11:58:53.031401 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.031383 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 11:58:53.031498 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.031405 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 11:58:53.031498 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.031428 2570 state_mem.go:36] "Initialized new in-memory state store" Jul 15 11:58:53.032574 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.032556 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 11:58:53.033282 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.033267 2570 policy_none.go:49] "None policy: Start" Jul 15 11:58:53.033282 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.033285 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 11:58:53.033387 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.033296 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 11:58:53.033654 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.033634 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasSufficientMemory" Jul 15 11:58:53.033758 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.033668 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 11:58:53.033758 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.033682 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasSufficientPID" Jul 15 11:58:53.034500 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.034487 2570 policy_none.go:47] "Start" Jul 15 11:58:53.074574 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.074370 2570 manager.go:337] "Starting Device Plugin manager" Jul 15 11:58:53.087142 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.074619 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 11:58:53.087142 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.074636 2570 server.go:87] "Starting device plugin registration server" Jul 15 11:58:53.087142 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.074906 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 11:58:53.087142 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.074918 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 11:58:53.087142 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.075004 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 11:58:53.087142 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.075077 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 11:58:53.087142 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.075083 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 11:58:53.087142 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.075664 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 11:58:53.087142 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.075897 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:53.087142 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.084143 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 11:58:53.087142 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.085423 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 11:58:53.087142 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.085456 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 11:58:53.087142 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.085478 2570 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 11:58:53.087142 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.085509 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 11:58:53.088128 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.088113 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 11:58:53.175831 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.175709 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 11:58:53.176898 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.176877 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasSufficientMemory" Jul 15 11:58:53.177014 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.176918 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 11:58:53.177014 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.176929 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasSufficientPID" Jul 15 11:58:53.177014 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.176952 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.186317 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.186293 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-53.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal"] Jul 15 11:58:53.186317 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.186311 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.186519 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.186335 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-53.ec2.internal\": node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:53.186519 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.186358 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 11:58:53.187286 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.187268 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasSufficientMemory" Jul 15 11:58:53.187380 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.187305 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 11:58:53.187380 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.187320 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasSufficientPID" Jul 15 11:58:53.188755 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.188721 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 11:58:53.188850 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.188830 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.188899 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.188876 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 11:58:53.189541 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.189520 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasSufficientMemory" Jul 15 11:58:53.189628 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.189547 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 11:58:53.189628 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.189523 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasSufficientMemory" Jul 15 11:58:53.189628 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.189561 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasSufficientPID" Jul 15 11:58:53.189628 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.189578 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 11:58:53.189628 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.189590 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasSufficientPID" Jul 15 11:58:53.190963 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.190946 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.191046 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.190979 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 11:58:53.191686 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.191669 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasSufficientMemory" Jul 15 11:58:53.191774 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.191701 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 11:58:53.191774 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.191713 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeHasSufficientPID" Jul 15 11:58:53.204266 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.204245 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:53.212488 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.212464 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21e4333f5fd0a8a3d223f0d7258f6334-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal\" (UID: \"21e4333f5fd0a8a3d223f0d7258f6334\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.212578 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.212495 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72a235d1a8bfb42e6972f74d75275638-config\") pod \"kube-apiserver-proxy-ip-10-0-131-53.ec2.internal\" (UID: \"72a235d1a8bfb42e6972f74d75275638\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.212578 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.212527 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21e4333f5fd0a8a3d223f0d7258f6334-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal\" (UID: \"21e4333f5fd0a8a3d223f0d7258f6334\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.217684 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.217664 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-53.ec2.internal\" not found" node="ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.222197 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.222178 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-53.ec2.internal\" not found" node="ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.304812 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.304779 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:53.313226 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.313194 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72a235d1a8bfb42e6972f74d75275638-config\") pod \"kube-apiserver-proxy-ip-10-0-131-53.ec2.internal\" (UID: \"72a235d1a8bfb42e6972f74d75275638\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.313353 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.313237 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21e4333f5fd0a8a3d223f0d7258f6334-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal\" (UID: \"21e4333f5fd0a8a3d223f0d7258f6334\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.313353 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.313263 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21e4333f5fd0a8a3d223f0d7258f6334-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal\" (UID: \"21e4333f5fd0a8a3d223f0d7258f6334\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.313353 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.313303 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72a235d1a8bfb42e6972f74d75275638-config\") pod \"kube-apiserver-proxy-ip-10-0-131-53.ec2.internal\" (UID: \"72a235d1a8bfb42e6972f74d75275638\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.313353 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.313318 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/21e4333f5fd0a8a3d223f0d7258f6334-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal\" (UID: \"21e4333f5fd0a8a3d223f0d7258f6334\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.313353 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.313319 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21e4333f5fd0a8a3d223f0d7258f6334-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal\" (UID: \"21e4333f5fd0a8a3d223f0d7258f6334\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.404896 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.404839 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:53.505872 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.505790 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:53.520401 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.520378 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.526031 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.526012 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal" Jul 15 11:58:53.606598 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.606556 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:53.707137 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.707083 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:53.807752 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.807661 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:53.843164 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.843139 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 11:58:53.908316 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:53.908272 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:53.936764 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.936725 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 11:58:53.937254 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.936884 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 11:58:53.937254 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.936929 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 11:58:53.937254 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:53.936927 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 11:58:54.008661 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:54.008622 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:54.009956 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.009927 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 11:53:53 +0000 UTC" deadline="2028-03-15 04:38:12.861002663 +0000 UTC" Jul 15 11:58:54.009956 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.009953 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14608h39m18.851052311s" Jul 15 11:58:54.109370 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:54.109278 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:54.209794 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:54.209758 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:54.267924 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:54.267880 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e4333f5fd0a8a3d223f0d7258f6334.slice/crio-4ae8e3eae4768328c5ab2f20db0c90627edc55600d85726a389eef99fc7dc3f4 WatchSource:0}: Error finding container 4ae8e3eae4768328c5ab2f20db0c90627edc55600d85726a389eef99fc7dc3f4: Status 404 returned error can't find the container with id 4ae8e3eae4768328c5ab2f20db0c90627edc55600d85726a389eef99fc7dc3f4 Jul 15 11:58:54.272552 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.272534 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 11:58:54.311860 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:54.311823 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:54.416804 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:54.416718 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-53.ec2.internal\" not found" Jul 15 11:58:54.436229 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.436195 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 11:58:54.510888 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.510850 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-53.ec2.internal" Jul 15 11:58:54.521197 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.521174 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 11:58:54.522491 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.522476 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal" Jul 15 11:58:54.534445 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.534419 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 11:58:54.728219 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:54.728194 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a235d1a8bfb42e6972f74d75275638.slice/crio-fbd48ce588e78cb010d0d814111bd084d8f6ac4a8a1517672e8b61557fde885d WatchSource:0}: Error finding container fbd48ce588e78cb010d0d814111bd084d8f6ac4a8a1517672e8b61557fde885d: Status 404 returned error can't find the container with id fbd48ce588e78cb010d0d814111bd084d8f6ac4a8a1517672e8b61557fde885d Jul 15 11:58:54.764605 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.764570 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 11:58:54.985122 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.985020 2570 apiserver.go:52] "Watching apiserver" Jul 15 11:58:54.994215 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.994176 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 11:58:54.994555 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.994533 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-ggb88","kube-system/konnectivity-agent-fr6q8","kube-system/kube-apiserver-proxy-ip-10-0-131-53.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal","openshift-multus/multus-nzwbz","openshift-network-diagnostics/network-check-target-pd45b","openshift-network-operator/iptables-alerter-ckhmn","openshift-cluster-node-tuning-operator/tuned-dzq6d","openshift-dns/node-resolver-9zcb8","openshift-image-registry/node-ca-snrv8","openshift-multus/multus-additional-cni-plugins-2pbtb","openshift-multus/network-metrics-daemon-l4d6j","openshift-ovn-kubernetes/ovnkube-node-7kxg9"] Jul 15 11:58:54.996316 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.996292 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:58:54.996425 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:54.996364 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:58:54.997448 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.997429 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-snrv8" Jul 15 11:58:54.999057 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.999036 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:54.999446 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.999420 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 11:58:54.999769 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.999745 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 11:58:55.000005 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:54.999988 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 11:58:55.000087 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.000036 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fxxl9\"" Jul 15 11:58:55.000795 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.000771 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.001244 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.001220 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 11:58:55.001244 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.001240 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 11:58:55.001468 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.001448 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 11:58:55.001662 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.001641 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 11:58:55.001763 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.001712 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w2p9m\"" Jul 15 11:58:55.001841 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.001763 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 11:58:55.002012 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.001993 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 11:58:55.002324 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.002301 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ckhmn" Jul 15 11:58:55.002377 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.002364 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 11:58:55.002437 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.002374 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rp8l6\"" Jul 15 11:58:55.002652 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.002616 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 11:58:55.002905 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.002886 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 11:58:55.003784 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.003768 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.004268 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.004247 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qgpfq\"" Jul 15 11:58:55.004660 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.004647 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 11:58:55.004782 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.004759 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 11:58:55.004902 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.004859 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 11:58:55.005594 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.005566 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.005701 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.005625 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 11:58:55.006127 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.005850 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 11:58:55.006127 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.005912 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 11:58:55.006127 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.005925 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-59sdv\"" Jul 15 11:58:55.006284 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.006149 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 11:58:55.007006 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.006992 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:58:55.007068 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.007051 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:58:55.008086 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.008067 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-znv67\"" Jul 15 11:58:55.008086 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.008080 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 11:58:55.008253 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.008150 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 11:58:55.008568 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.008549 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fr6q8" Jul 15 11:58:55.010023 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.010007 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9zcb8" Jul 15 11:58:55.010023 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.010014 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 11:58:55.010337 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.010314 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 11:58:55.010515 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.010497 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 11:58:55.010709 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.010693 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4gf2x\"" Jul 15 11:58:55.011499 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.011481 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:58:55.011579 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.011542 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:58:55.011813 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.011798 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-t6llw\"" Jul 15 11:58:55.012003 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.011989 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 11:58:55.012125 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.012107 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 11:58:55.012990 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.012969 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.014977 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.014958 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 11:58:55.015047 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.014964 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 11:58:55.015047 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.015006 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tl5r8\"" Jul 15 11:58:55.022525 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022495 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-registration-dir\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.022646 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022537 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmgrl\" (UniqueName: \"kubernetes.io/projected/3a23f0bc-7022-4e72-b3cf-0a63b6e681f8-kube-api-access-rmgrl\") pod \"node-resolver-9zcb8\" (UID: \"3a23f0bc-7022-4e72-b3cf-0a63b6e681f8\") " pod="openshift-dns/node-resolver-9zcb8" Jul 15 11:58:55.022646 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022564 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpxjs\" (UniqueName: \"kubernetes.io/projected/76443aa4-c20f-4f32-8bbd-f88a776b0b34-kube-api-access-rpxjs\") pod \"network-metrics-daemon-l4d6j\" (UID: \"76443aa4-c20f-4f32-8bbd-f88a776b0b34\") " pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:58:55.022646 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022588 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-lib-modules\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.022646 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022613 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drxw7\" (UniqueName: \"kubernetes.io/projected/e60c06f6-afda-41ff-b0af-eb7a11e513a3-kube-api-access-drxw7\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.022646 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022637 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-systemd-units\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.022951 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022660 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-cni-netd\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.022951 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022681 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-multus-cni-dir\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.022951 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022713 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-hostroot\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.022951 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022734 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a23f0bc-7022-4e72-b3cf-0a63b6e681f8-hosts-file\") pod \"node-resolver-9zcb8\" (UID: \"3a23f0bc-7022-4e72-b3cf-0a63b6e681f8\") " pod="openshift-dns/node-resolver-9zcb8" Jul 15 11:58:55.022951 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022761 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-kubernetes\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.022951 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022802 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-sysctl-d\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.022951 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022823 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e60c06f6-afda-41ff-b0af-eb7a11e513a3-tmp\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.022951 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022822 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 11:58:55.022951 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022847 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtlz\" (UniqueName: \"kubernetes.io/projected/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-kube-api-access-9mtlz\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.022951 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022871 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-multus-socket-dir-parent\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.022951 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022917 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-tuned\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.023402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.022979 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-run-netns\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.023402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023016 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-device-dir\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.023402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023044 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-run-netns\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.023402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023079 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac9ea448-c0c5-4782-833e-e767a709de84-multus-daemon-config\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.023402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023111 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs\") pod \"network-metrics-daemon-l4d6j\" (UID: \"76443aa4-c20f-4f32-8bbd-f88a776b0b34\") " pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:58:55.023402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023146 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e507f9d6-cd28-4d22-9dfe-5869963a8c48-serviceca\") pod \"node-ca-snrv8\" (UID: \"e507f9d6-cd28-4d22-9dfe-5869963a8c48\") " pod="openshift-image-registry/node-ca-snrv8" Jul 15 11:58:55.023402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023177 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-run\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.023402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023202 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e507f9d6-cd28-4d22-9dfe-5869963a8c48-host\") pod \"node-ca-snrv8\" (UID: \"e507f9d6-cd28-4d22-9dfe-5869963a8c48\") " pod="openshift-image-registry/node-ca-snrv8" Jul 15 11:58:55.023402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023242 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-slash\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.023402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023283 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-var-lib-openvswitch\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.023402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023312 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-run-ovn\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.023402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023338 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52d7772a-574c-4960-9541-e138399719f3-ovn-node-metrics-cert\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.023402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023366 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88807a13-409e-4b95-b399-43ed30674850-konnectivity-ca\") pod \"konnectivity-agent-fr6q8\" (UID: \"88807a13-409e-4b95-b399-43ed30674850\") " pod="kube-system/konnectivity-agent-fr6q8" Jul 15 11:58:55.023402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023391 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-systemd\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023415 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-sys\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023441 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-kubelet\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023467 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdslb\" (UniqueName: \"kubernetes.io/projected/52d7772a-574c-4960-9541-e138399719f3-kube-api-access-vdslb\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023504 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-system-cni-dir\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023529 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-run-k8s-cni-cncf-io\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023555 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-etc-kubernetes\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023580 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpcbv\" (UniqueName: \"kubernetes.io/projected/ac9ea448-c0c5-4782-833e-e767a709de84-kube-api-access-bpcbv\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023605 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/86361831-462f-4f0d-b679-adefb3cd226c-iptables-alerter-script\") pod \"iptables-alerter-ckhmn\" (UID: \"86361831-462f-4f0d-b679-adefb3cd226c\") " pod="openshift-network-operator/iptables-alerter-ckhmn" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023631 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023672 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-cni-bin\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023697 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-socket-dir\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023722 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-var-lib-kubelet\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023748 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-run-multus-certs\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023779 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86361831-462f-4f0d-b679-adefb3cd226c-host-slash\") pod \"iptables-alerter-ckhmn\" (UID: \"86361831-462f-4f0d-b679-adefb3cd226c\") " pod="openshift-network-operator/iptables-alerter-ckhmn" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023805 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-node-log\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.023998 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023826 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023855 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-var-lib-cni-multus\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023876 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-sysconfig\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023902 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-run-ovn-kubernetes\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023917 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-multus-conf-dir\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023931 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgqlz\" (UniqueName: \"kubernetes.io/projected/e507f9d6-cd28-4d22-9dfe-5869963a8c48-kube-api-access-cgqlz\") pod \"node-ca-snrv8\" (UID: \"e507f9d6-cd28-4d22-9dfe-5869963a8c48\") " pod="openshift-image-registry/node-ca-snrv8" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023953 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8rw8\" (UniqueName: \"kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8\") pod \"network-check-target-pd45b\" (UID: \"0c8aff4d-50ce-4f75-84fb-786fe725d46e\") " pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023968 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-etc-selinux\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.023982 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-modprobe-d\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024004 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-run-openvswitch\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024028 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-log-socket\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024053 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-cnibin\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024073 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-os-release\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024107 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqhpn\" (UniqueName: \"kubernetes.io/projected/86361831-462f-4f0d-b679-adefb3cd226c-kube-api-access-fqhpn\") pod \"iptables-alerter-ckhmn\" (UID: \"86361831-462f-4f0d-b679-adefb3cd226c\") " pod="openshift-network-operator/iptables-alerter-ckhmn" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024136 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-sysctl-conf\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024165 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/977f5c84-a6ea-4927-93ad-9dd0e23853a8-dbus\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:58:55.024617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024191 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52d7772a-574c-4960-9541-e138399719f3-ovnkube-config\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.025279 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024212 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-var-lib-cni-bin\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.025279 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024227 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a23f0bc-7022-4e72-b3cf-0a63b6e681f8-tmp-dir\") pod \"node-resolver-9zcb8\" (UID: \"3a23f0bc-7022-4e72-b3cf-0a63b6e681f8\") " pod="openshift-dns/node-resolver-9zcb8" Jul 15 11:58:55.025279 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024242 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-var-lib-kubelet\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.025279 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024260 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-etc-openvswitch\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.025279 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024282 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.025279 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024305 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-sys-fs\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.025279 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024327 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac9ea448-c0c5-4782-833e-e767a709de84-cni-binary-copy\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.025279 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024344 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88807a13-409e-4b95-b399-43ed30674850-agent-certs\") pod \"konnectivity-agent-fr6q8\" (UID: \"88807a13-409e-4b95-b399-43ed30674850\") " pod="kube-system/konnectivity-agent-fr6q8" Jul 15 11:58:55.025279 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024373 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-host\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.025279 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024397 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/977f5c84-a6ea-4927-93ad-9dd0e23853a8-kubelet-config\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:58:55.025279 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024413 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-run-systemd\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.025279 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024428 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52d7772a-574c-4960-9541-e138399719f3-env-overrides\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.025279 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.024448 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52d7772a-574c-4960-9541-e138399719f3-ovnkube-script-lib\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.047932 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.047902 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bd8dt" Jul 15 11:58:55.056580 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.056560 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bd8dt" Jul 15 11:58:55.090108 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.090042 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-53.ec2.internal" event={"ID":"72a235d1a8bfb42e6972f74d75275638","Type":"ContainerStarted","Data":"fbd48ce588e78cb010d0d814111bd084d8f6ac4a8a1517672e8b61557fde885d"} Jul 15 11:58:55.090991 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.090969 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal" event={"ID":"21e4333f5fd0a8a3d223f0d7258f6334","Type":"ContainerStarted","Data":"4ae8e3eae4768328c5ab2f20db0c90627edc55600d85726a389eef99fc7dc3f4"} Jul 15 11:58:55.111285 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.111243 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 11:58:55.124877 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.124849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac9ea448-c0c5-4782-833e-e767a709de84-multus-daemon-config\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.124877 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.124883 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs\") pod \"network-metrics-daemon-l4d6j\" (UID: \"76443aa4-c20f-4f32-8bbd-f88a776b0b34\") " pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:58:55.125113 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.124901 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e507f9d6-cd28-4d22-9dfe-5869963a8c48-serviceca\") pod \"node-ca-snrv8\" (UID: \"e507f9d6-cd28-4d22-9dfe-5869963a8c48\") " pod="openshift-image-registry/node-ca-snrv8" Jul 15 11:58:55.125113 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.124917 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-run\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.125113 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.124938 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13a16a94-e9e0-4c90-9490-694591b8d873-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.125113 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.124970 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e507f9d6-cd28-4d22-9dfe-5869963a8c48-host\") pod \"node-ca-snrv8\" (UID: \"e507f9d6-cd28-4d22-9dfe-5869963a8c48\") " pod="openshift-image-registry/node-ca-snrv8" Jul 15 11:58:55.125113 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125018 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e507f9d6-cd28-4d22-9dfe-5869963a8c48-host\") pod \"node-ca-snrv8\" (UID: \"e507f9d6-cd28-4d22-9dfe-5869963a8c48\") " pod="openshift-image-registry/node-ca-snrv8" Jul 15 11:58:55.125113 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.125039 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 11:58:55.125113 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125036 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13a16a94-e9e0-4c90-9490-694591b8d873-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.125113 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125036 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-run\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.125135 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs podName:76443aa4-c20f-4f32-8bbd-f88a776b0b34 nodeName:}" failed. No retries permitted until 2026-07-15 11:58:55.625097852 +0000 UTC m=+3.038626182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs") pod "network-metrics-daemon-l4d6j" (UID: "76443aa4-c20f-4f32-8bbd-f88a776b0b34") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125230 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-slash\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125285 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-var-lib-openvswitch\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125303 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-slash\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125312 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-run-ovn\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125338 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52d7772a-574c-4960-9541-e138399719f3-ovn-node-metrics-cert\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125347 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-var-lib-openvswitch\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125365 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88807a13-409e-4b95-b399-43ed30674850-konnectivity-ca\") pod \"konnectivity-agent-fr6q8\" (UID: \"88807a13-409e-4b95-b399-43ed30674850\") " pod="kube-system/konnectivity-agent-fr6q8" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125378 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-run-ovn\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125390 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-systemd\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125414 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-sys\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125440 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13a16a94-e9e0-4c90-9490-694591b8d873-system-cni-dir\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125454 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e507f9d6-cd28-4d22-9dfe-5869963a8c48-serviceca\") pod \"node-ca-snrv8\" (UID: \"e507f9d6-cd28-4d22-9dfe-5869963a8c48\") " pod="openshift-image-registry/node-ca-snrv8" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125469 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-kubelet\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125479 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-systemd\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125494 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdslb\" (UniqueName: \"kubernetes.io/projected/52d7772a-574c-4960-9541-e138399719f3-kube-api-access-vdslb\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.125520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125510 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-sys\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125520 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-system-cni-dir\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125552 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-kubelet\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125557 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-run-k8s-cni-cncf-io\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125596 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-run-k8s-cni-cncf-io\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125626 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-etc-kubernetes\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125646 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-system-cni-dir\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125653 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpcbv\" (UniqueName: \"kubernetes.io/projected/ac9ea448-c0c5-4782-833e-e767a709de84-kube-api-access-bpcbv\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125708 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/86361831-462f-4f0d-b679-adefb3cd226c-iptables-alerter-script\") pod \"iptables-alerter-ckhmn\" (UID: \"86361831-462f-4f0d-b679-adefb3cd226c\") " pod="openshift-network-operator/iptables-alerter-ckhmn" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125714 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-etc-kubernetes\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125704 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125742 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125771 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-cni-bin\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125799 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-socket-dir\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125848 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-var-lib-kubelet\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.125869 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125873 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-run-multus-certs\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125900 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86361831-462f-4f0d-b679-adefb3cd226c-host-slash\") pod \"iptables-alerter-ckhmn\" (UID: \"86361831-462f-4f0d-b679-adefb3cd226c\") " pod="openshift-network-operator/iptables-alerter-ckhmn" Jul 15 11:58:55.126392 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.125931 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret podName:977f5c84-a6ea-4927-93ad-9dd0e23853a8 nodeName:}" failed. No retries permitted until 2026-07-15 11:58:55.625911382 +0000 UTC m=+3.039439713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret") pod "global-pull-secret-syncer-ggb88" (UID: "977f5c84-a6ea-4927-93ad-9dd0e23853a8") : object "kube-system"/"original-pull-secret" not registered Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125946 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88807a13-409e-4b95-b399-43ed30674850-konnectivity-ca\") pod \"konnectivity-agent-fr6q8\" (UID: \"88807a13-409e-4b95-b399-43ed30674850\") " pod="kube-system/konnectivity-agent-fr6q8" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125954 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/13a16a94-e9e0-4c90-9490-694591b8d873-os-release\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125949 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86361831-462f-4f0d-b679-adefb3cd226c-host-slash\") pod \"iptables-alerter-ckhmn\" (UID: \"86361831-462f-4f0d-b679-adefb3cd226c\") " pod="openshift-network-operator/iptables-alerter-ckhmn" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125978 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-cni-bin\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.125987 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27db2\" (UniqueName: \"kubernetes.io/projected/13a16a94-e9e0-4c90-9490-694591b8d873-kube-api-access-27db2\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126035 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-run-multus-certs\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126041 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-var-lib-kubelet\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126075 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-node-log\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126097 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126125 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-var-lib-cni-multus\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126140 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-socket-dir\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126147 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-node-log\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126146 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-sysconfig\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126175 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126191 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-run-ovn-kubernetes\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126214 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-multus-conf-dir\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.127304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126222 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-run-ovn-kubernetes\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126227 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-var-lib-cni-multus\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126235 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgqlz\" (UniqueName: \"kubernetes.io/projected/e507f9d6-cd28-4d22-9dfe-5869963a8c48-kube-api-access-cgqlz\") pod \"node-ca-snrv8\" (UID: \"e507f9d6-cd28-4d22-9dfe-5869963a8c48\") " pod="openshift-image-registry/node-ca-snrv8" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126193 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-sysconfig\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126258 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/86361831-462f-4f0d-b679-adefb3cd226c-iptables-alerter-script\") pod \"iptables-alerter-ckhmn\" (UID: \"86361831-462f-4f0d-b679-adefb3cd226c\") " pod="openshift-network-operator/iptables-alerter-ckhmn" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126265 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-multus-conf-dir\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126259 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rw8\" (UniqueName: \"kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8\") pod \"network-check-target-pd45b\" (UID: \"0c8aff4d-50ce-4f75-84fb-786fe725d46e\") " pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126308 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-etc-selinux\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126333 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-modprobe-d\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126441 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-modprobe-d\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126440 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-etc-selinux\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126463 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-run-openvswitch\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126504 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-log-socket\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126506 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-run-openvswitch\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126551 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-log-socket\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126550 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-cnibin\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126606 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-cnibin\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126641 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-os-release\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.128216 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126654 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ac9ea448-c0c5-4782-833e-e767a709de84-multus-daemon-config\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126666 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqhpn\" (UniqueName: \"kubernetes.io/projected/86361831-462f-4f0d-b679-adefb3cd226c-kube-api-access-fqhpn\") pod \"iptables-alerter-ckhmn\" (UID: \"86361831-462f-4f0d-b679-adefb3cd226c\") " pod="openshift-network-operator/iptables-alerter-ckhmn" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126678 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-os-release\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126688 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-sysctl-conf\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126710 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/977f5c84-a6ea-4927-93ad-9dd0e23853a8-dbus\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126731 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52d7772a-574c-4960-9541-e138399719f3-ovnkube-config\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126772 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-var-lib-cni-bin\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126793 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a23f0bc-7022-4e72-b3cf-0a63b6e681f8-tmp-dir\") pod \"node-resolver-9zcb8\" (UID: \"3a23f0bc-7022-4e72-b3cf-0a63b6e681f8\") " pod="openshift-dns/node-resolver-9zcb8" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126830 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-var-lib-kubelet\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126846 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/977f5c84-a6ea-4927-93ad-9dd0e23853a8-dbus\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126851 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-etc-openvswitch\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126877 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126883 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-etc-openvswitch\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126894 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-sys-fs\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126914 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac9ea448-c0c5-4782-833e-e767a709de84-cni-binary-copy\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126946 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-var-lib-kubelet\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.126994 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-sys-fs\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.128950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127015 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127136 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/52d7772a-574c-4960-9541-e138399719f3-ovnkube-config\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127174 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-var-lib-cni-bin\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127195 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88807a13-409e-4b95-b399-43ed30674850-agent-certs\") pod \"konnectivity-agent-fr6q8\" (UID: \"88807a13-409e-4b95-b399-43ed30674850\") " pod="kube-system/konnectivity-agent-fr6q8" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127212 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-host\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127229 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/977f5c84-a6ea-4927-93ad-9dd0e23853a8-kubelet-config\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127245 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-run-systemd\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127261 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52d7772a-574c-4960-9541-e138399719f3-env-overrides\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127278 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52d7772a-574c-4960-9541-e138399719f3-ovnkube-script-lib\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127269 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-sysctl-conf\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127297 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-registration-dir\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127307 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ac9ea448-c0c5-4782-833e-e767a709de84-cni-binary-copy\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127311 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a23f0bc-7022-4e72-b3cf-0a63b6e681f8-tmp-dir\") pod \"node-resolver-9zcb8\" (UID: \"3a23f0bc-7022-4e72-b3cf-0a63b6e681f8\") " pod="openshift-dns/node-resolver-9zcb8" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127326 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-registration-dir\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127333 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmgrl\" (UniqueName: \"kubernetes.io/projected/3a23f0bc-7022-4e72-b3cf-0a63b6e681f8-kube-api-access-rmgrl\") pod \"node-resolver-9zcb8\" (UID: \"3a23f0bc-7022-4e72-b3cf-0a63b6e681f8\") " pod="openshift-dns/node-resolver-9zcb8" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127358 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-run-systemd\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127359 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/977f5c84-a6ea-4927-93ad-9dd0e23853a8-kubelet-config\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:58:55.129561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127363 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpxjs\" (UniqueName: \"kubernetes.io/projected/76443aa4-c20f-4f32-8bbd-f88a776b0b34-kube-api-access-rpxjs\") pod \"network-metrics-daemon-l4d6j\" (UID: \"76443aa4-c20f-4f32-8bbd-f88a776b0b34\") " pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127399 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-host\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127487 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-lib-modules\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127518 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxw7\" (UniqueName: \"kubernetes.io/projected/e60c06f6-afda-41ff-b0af-eb7a11e513a3-kube-api-access-drxw7\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127545 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-systemd-units\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127573 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-cni-netd\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127597 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-multus-cni-dir\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127622 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-hostroot\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127651 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a23f0bc-7022-4e72-b3cf-0a63b6e681f8-hosts-file\") pod \"node-resolver-9zcb8\" (UID: \"3a23f0bc-7022-4e72-b3cf-0a63b6e681f8\") " pod="openshift-dns/node-resolver-9zcb8" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127677 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-kubernetes\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127702 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-sysctl-d\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127721 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/52d7772a-574c-4960-9541-e138399719f3-ovnkube-script-lib\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127734 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-cni-netd\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127727 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e60c06f6-afda-41ff-b0af-eb7a11e513a3-tmp\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127790 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtlz\" (UniqueName: \"kubernetes.io/projected/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-kube-api-access-9mtlz\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127828 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a23f0bc-7022-4e72-b3cf-0a63b6e681f8-hosts-file\") pod \"node-resolver-9zcb8\" (UID: \"3a23f0bc-7022-4e72-b3cf-0a63b6e681f8\") " pod="openshift-dns/node-resolver-9zcb8" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127813 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-systemd-units\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127835 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-multus-socket-dir-parent\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.130130 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127877 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-sysctl-d\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127886 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-tuned\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127918 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13a16a94-e9e0-4c90-9490-694591b8d873-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127924 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-kubernetes\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127938 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-multus-cni-dir\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127947 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-run-netns\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127969 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-hostroot\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127974 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13a16a94-e9e0-4c90-9490-694591b8d873-cnibin\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.128022 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e60c06f6-afda-41ff-b0af-eb7a11e513a3-lib-modules\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.127888 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-multus-socket-dir-parent\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.128062 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/52d7772a-574c-4960-9541-e138399719f3-host-run-netns\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.128093 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13a16a94-e9e0-4c90-9490-694591b8d873-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.128318 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-device-dir\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.128345 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-run-netns\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.128367 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/52d7772a-574c-4960-9541-e138399719f3-env-overrides\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.128432 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-device-dir\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.128419 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac9ea448-c0c5-4782-833e-e767a709de84-host-run-netns\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.130847 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.130077 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e60c06f6-afda-41ff-b0af-eb7a11e513a3-tmp\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.131349 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.130208 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/52d7772a-574c-4960-9541-e138399719f3-ovn-node-metrics-cert\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.131349 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.130275 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88807a13-409e-4b95-b399-43ed30674850-agent-certs\") pod \"konnectivity-agent-fr6q8\" (UID: \"88807a13-409e-4b95-b399-43ed30674850\") " pod="kube-system/konnectivity-agent-fr6q8" Jul 15 11:58:55.131349 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.130398 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e60c06f6-afda-41ff-b0af-eb7a11e513a3-etc-tuned\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.139921 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.139588 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 11:58:55.139921 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.139617 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 11:58:55.139921 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.139633 2570 projected.go:196] Error preparing data for projected volume kube-api-access-f8rw8 for pod openshift-network-diagnostics/network-check-target-pd45b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 11:58:55.139921 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.139704 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8 podName:0c8aff4d-50ce-4f75-84fb-786fe725d46e nodeName:}" failed. No retries permitted until 2026-07-15 11:58:55.63968275 +0000 UTC m=+3.053211107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f8rw8" (UniqueName: "kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8") pod "network-check-target-pd45b" (UID: "0c8aff4d-50ce-4f75-84fb-786fe725d46e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 11:58:55.140291 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.140175 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpcbv\" (UniqueName: \"kubernetes.io/projected/ac9ea448-c0c5-4782-833e-e767a709de84-kube-api-access-bpcbv\") pod \"multus-nzwbz\" (UID: \"ac9ea448-c0c5-4782-833e-e767a709de84\") " pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.141949 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.141916 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgqlz\" (UniqueName: \"kubernetes.io/projected/e507f9d6-cd28-4d22-9dfe-5869963a8c48-kube-api-access-cgqlz\") pod \"node-ca-snrv8\" (UID: \"e507f9d6-cd28-4d22-9dfe-5869963a8c48\") " pod="openshift-image-registry/node-ca-snrv8" Jul 15 11:58:55.142496 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.142469 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmgrl\" (UniqueName: \"kubernetes.io/projected/3a23f0bc-7022-4e72-b3cf-0a63b6e681f8-kube-api-access-rmgrl\") pod \"node-resolver-9zcb8\" (UID: \"3a23f0bc-7022-4e72-b3cf-0a63b6e681f8\") " pod="openshift-dns/node-resolver-9zcb8" Jul 15 11:58:55.142604 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.142582 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpxjs\" (UniqueName: \"kubernetes.io/projected/76443aa4-c20f-4f32-8bbd-f88a776b0b34-kube-api-access-rpxjs\") pod \"network-metrics-daemon-l4d6j\" (UID: \"76443aa4-c20f-4f32-8bbd-f88a776b0b34\") " pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:58:55.142722 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.142703 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqhpn\" (UniqueName: \"kubernetes.io/projected/86361831-462f-4f0d-b679-adefb3cd226c-kube-api-access-fqhpn\") pod \"iptables-alerter-ckhmn\" (UID: \"86361831-462f-4f0d-b679-adefb3cd226c\") " pod="openshift-network-operator/iptables-alerter-ckhmn" Jul 15 11:58:55.142965 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.142947 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdslb\" (UniqueName: \"kubernetes.io/projected/52d7772a-574c-4960-9541-e138399719f3-kube-api-access-vdslb\") pod \"ovnkube-node-7kxg9\" (UID: \"52d7772a-574c-4960-9541-e138399719f3\") " pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.143046 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.143011 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtlz\" (UniqueName: \"kubernetes.io/projected/cc7e18cf-ef4d-4dd2-8d45-efd745c4b358-kube-api-access-9mtlz\") pod \"aws-ebs-csi-driver-node-q2lp2\" (UID: \"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.144215 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.144197 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxw7\" (UniqueName: \"kubernetes.io/projected/e60c06f6-afda-41ff-b0af-eb7a11e513a3-kube-api-access-drxw7\") pod \"tuned-dzq6d\" (UID: \"e60c06f6-afda-41ff-b0af-eb7a11e513a3\") " pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.228672 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.228634 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/13a16a94-e9e0-4c90-9490-694591b8d873-os-release\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.228672 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.228669 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27db2\" (UniqueName: \"kubernetes.io/projected/13a16a94-e9e0-4c90-9490-694591b8d873-kube-api-access-27db2\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.228940 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.228715 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13a16a94-e9e0-4c90-9490-694591b8d873-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.228940 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.228733 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13a16a94-e9e0-4c90-9490-694591b8d873-cnibin\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.228940 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.228749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13a16a94-e9e0-4c90-9490-694591b8d873-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.228940 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.228788 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/13a16a94-e9e0-4c90-9490-694591b8d873-os-release\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.228940 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.228800 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13a16a94-e9e0-4c90-9490-694591b8d873-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.228940 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.228836 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13a16a94-e9e0-4c90-9490-694591b8d873-cnibin\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.228940 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.228867 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13a16a94-e9e0-4c90-9490-694591b8d873-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.228940 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.228902 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13a16a94-e9e0-4c90-9490-694591b8d873-system-cni-dir\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.228940 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.228919 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13a16a94-e9e0-4c90-9490-694591b8d873-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.229304 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.228968 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13a16a94-e9e0-4c90-9490-694591b8d873-system-cni-dir\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.229401 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.229382 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13a16a94-e9e0-4c90-9490-694591b8d873-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.229401 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.229389 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13a16a94-e9e0-4c90-9490-694591b8d873-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.229465 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.229450 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13a16a94-e9e0-4c90-9490-694591b8d873-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.240150 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.240083 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27db2\" (UniqueName: \"kubernetes.io/projected/13a16a94-e9e0-4c90-9490-694591b8d873-kube-api-access-27db2\") pod \"multus-additional-cni-plugins-2pbtb\" (UID: \"13a16a94-e9e0-4c90-9490-694591b8d873\") " pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.309509 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.309462 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-snrv8" Jul 15 11:58:55.316070 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.316046 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:58:55.319250 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:55.319220 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode507f9d6_cd28_4d22_9dfe_5869963a8c48.slice/crio-70f1424d1b3916379222c6ea84073292c61ca76a9f05bf704a8384d3faf7cb4f WatchSource:0}: Error finding container 70f1424d1b3916379222c6ea84073292c61ca76a9f05bf704a8384d3faf7cb4f: Status 404 returned error can't find the container with id 70f1424d1b3916379222c6ea84073292c61ca76a9f05bf704a8384d3faf7cb4f Jul 15 11:58:55.326722 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.326699 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" Jul 15 11:58:55.327033 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:55.327003 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d7772a_574c_4960_9541_e138399719f3.slice/crio-e4a5b3c2cb2f2da0d16b53b4262d3abfb2a969962cce42e669b80bd8957c1a4b WatchSource:0}: Error finding container e4a5b3c2cb2f2da0d16b53b4262d3abfb2a969962cce42e669b80bd8957c1a4b: Status 404 returned error can't find the container with id e4a5b3c2cb2f2da0d16b53b4262d3abfb2a969962cce42e669b80bd8957c1a4b Jul 15 11:58:55.332087 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.332064 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ckhmn" Jul 15 11:58:55.337580 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.337546 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nzwbz" Jul 15 11:58:55.338187 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:55.338146 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7e18cf_ef4d_4dd2_8d45_efd745c4b358.slice/crio-12f43729a9365ac9da55ff286958e3443ef06fd428f0a7b9cb94aee6a31f5f37 WatchSource:0}: Error finding container 12f43729a9365ac9da55ff286958e3443ef06fd428f0a7b9cb94aee6a31f5f37: Status 404 returned error can't find the container with id 12f43729a9365ac9da55ff286958e3443ef06fd428f0a7b9cb94aee6a31f5f37 Jul 15 11:58:55.343634 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.343608 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" Jul 15 11:58:55.345827 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:55.345602 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86361831_462f_4f0d_b679_adefb3cd226c.slice/crio-6bd401f7b668c69d4f84f2d65a1b4882187770f8435aa7eb0daf6f3f1d7b2f68 WatchSource:0}: Error finding container 6bd401f7b668c69d4f84f2d65a1b4882187770f8435aa7eb0daf6f3f1d7b2f68: Status 404 returned error can't find the container with id 6bd401f7b668c69d4f84f2d65a1b4882187770f8435aa7eb0daf6f3f1d7b2f68 Jul 15 11:58:55.349469 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.348549 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fr6q8" Jul 15 11:58:55.349777 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:55.349747 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9ea448_c0c5_4782_833e_e767a709de84.slice/crio-11017d5bceccf542247f671edc0aae6b303509e1bc01c24bb053f0aabed14719 WatchSource:0}: Error finding container 11017d5bceccf542247f671edc0aae6b303509e1bc01c24bb053f0aabed14719: Status 404 returned error can't find the container with id 11017d5bceccf542247f671edc0aae6b303509e1bc01c24bb053f0aabed14719 Jul 15 11:58:55.354867 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.354660 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9zcb8" Jul 15 11:58:55.359443 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.359418 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2pbtb" Jul 15 11:58:55.364062 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:55.364032 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88807a13_409e_4b95_b399_43ed30674850.slice/crio-5bc949ed2c006830e68799fd0db9165f5a8095fdf7d56ce029c16bee10424072 WatchSource:0}: Error finding container 5bc949ed2c006830e68799fd0db9165f5a8095fdf7d56ce029c16bee10424072: Status 404 returned error can't find the container with id 5bc949ed2c006830e68799fd0db9165f5a8095fdf7d56ce029c16bee10424072 Jul 15 11:58:55.368109 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:55.368082 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a23f0bc_7022_4e72_b3cf_0a63b6e681f8.slice/crio-be5e0c3c674b47d8162b131034a66f3d6fda53e523f17c1ff093940363a9f4a3 WatchSource:0}: Error finding container be5e0c3c674b47d8162b131034a66f3d6fda53e523f17c1ff093940363a9f4a3: Status 404 returned error can't find the container with id be5e0c3c674b47d8162b131034a66f3d6fda53e523f17c1ff093940363a9f4a3 Jul 15 11:58:55.372022 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:58:55.371992 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a16a94_e9e0_4c90_9490_694591b8d873.slice/crio-4e7c015e944c694842409bf36197aa5c1b0257967eb54e3c5786c0abeed74f18 WatchSource:0}: Error finding container 4e7c015e944c694842409bf36197aa5c1b0257967eb54e3c5786c0abeed74f18: Status 404 returned error can't find the container with id 4e7c015e944c694842409bf36197aa5c1b0257967eb54e3c5786c0abeed74f18 Jul 15 11:58:55.537845 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.537762 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 11:58:55.632069 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.632020 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs\") pod \"network-metrics-daemon-l4d6j\" (UID: \"76443aa4-c20f-4f32-8bbd-f88a776b0b34\") " pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:58:55.632242 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.632083 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:58:55.632242 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.632207 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 11:58:55.632242 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.632207 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 11:58:55.632382 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.632273 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret podName:977f5c84-a6ea-4927-93ad-9dd0e23853a8 nodeName:}" failed. No retries permitted until 2026-07-15 11:58:56.632254979 +0000 UTC m=+4.045783311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret") pod "global-pull-secret-syncer-ggb88" (UID: "977f5c84-a6ea-4927-93ad-9dd0e23853a8") : object "kube-system"/"original-pull-secret" not registered Jul 15 11:58:55.632382 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.632292 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs podName:76443aa4-c20f-4f32-8bbd-f88a776b0b34 nodeName:}" failed. No retries permitted until 2026-07-15 11:58:56.632282688 +0000 UTC m=+4.045811019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs") pod "network-metrics-daemon-l4d6j" (UID: "76443aa4-c20f-4f32-8bbd-f88a776b0b34") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 11:58:55.642314 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.642280 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 11:58:55.733334 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:55.733290 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rw8\" (UniqueName: \"kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8\") pod \"network-check-target-pd45b\" (UID: \"0c8aff4d-50ce-4f75-84fb-786fe725d46e\") " pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:58:55.733527 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.733459 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 11:58:55.733527 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.733491 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 11:58:55.733527 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.733506 2570 projected.go:196] Error preparing data for projected volume kube-api-access-f8rw8 for pod openshift-network-diagnostics/network-check-target-pd45b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 11:58:55.733800 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:55.733583 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8 podName:0c8aff4d-50ce-4f75-84fb-786fe725d46e nodeName:}" failed. No retries permitted until 2026-07-15 11:58:56.733560802 +0000 UTC m=+4.147089135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8rw8" (UniqueName: "kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8") pod "network-check-target-pd45b" (UID: "0c8aff4d-50ce-4f75-84fb-786fe725d46e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 11:58:56.059202 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:56.059146 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 11:53:55 +0000 UTC" deadline="2028-02-19 21:30:49.28667376 +0000 UTC" Jul 15 11:58:56.059202 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:56.059195 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14025h31m53.227484011s" Jul 15 11:58:56.095769 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:56.095712 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9zcb8" event={"ID":"3a23f0bc-7022-4e72-b3cf-0a63b6e681f8","Type":"ContainerStarted","Data":"be5e0c3c674b47d8162b131034a66f3d6fda53e523f17c1ff093940363a9f4a3"} Jul 15 11:58:56.097172 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:56.097133 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nzwbz" event={"ID":"ac9ea448-c0c5-4782-833e-e767a709de84","Type":"ContainerStarted","Data":"11017d5bceccf542247f671edc0aae6b303509e1bc01c24bb053f0aabed14719"} Jul 15 11:58:56.098534 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:56.098506 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ckhmn" event={"ID":"86361831-462f-4f0d-b679-adefb3cd226c","Type":"ContainerStarted","Data":"6bd401f7b668c69d4f84f2d65a1b4882187770f8435aa7eb0daf6f3f1d7b2f68"} Jul 15 11:58:56.101372 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:56.101346 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fr6q8" event={"ID":"88807a13-409e-4b95-b399-43ed30674850","Type":"ContainerStarted","Data":"5bc949ed2c006830e68799fd0db9165f5a8095fdf7d56ce029c16bee10424072"} Jul 15 11:58:56.102561 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:56.102532 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" event={"ID":"e60c06f6-afda-41ff-b0af-eb7a11e513a3","Type":"ContainerStarted","Data":"a1bfb2495be62f2404b5ddadd8d33cc913bd1f4ad2388e7e730cbe2846c9fc75"} Jul 15 11:58:56.103822 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:56.103789 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" event={"ID":"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358","Type":"ContainerStarted","Data":"12f43729a9365ac9da55ff286958e3443ef06fd428f0a7b9cb94aee6a31f5f37"} Jul 15 11:58:56.105209 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:56.105116 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" event={"ID":"52d7772a-574c-4960-9541-e138399719f3","Type":"ContainerStarted","Data":"e4a5b3c2cb2f2da0d16b53b4262d3abfb2a969962cce42e669b80bd8957c1a4b"} Jul 15 11:58:56.106320 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:56.106293 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-snrv8" event={"ID":"e507f9d6-cd28-4d22-9dfe-5869963a8c48","Type":"ContainerStarted","Data":"70f1424d1b3916379222c6ea84073292c61ca76a9f05bf704a8384d3faf7cb4f"} Jul 15 11:58:56.107401 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:56.107373 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtb" event={"ID":"13a16a94-e9e0-4c90-9490-694591b8d873","Type":"ContainerStarted","Data":"4e7c015e944c694842409bf36197aa5c1b0257967eb54e3c5786c0abeed74f18"} Jul 15 11:58:56.643504 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:56.643422 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs\") pod \"network-metrics-daemon-l4d6j\" (UID: \"76443aa4-c20f-4f32-8bbd-f88a776b0b34\") " pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:58:56.643504 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:56.643471 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:58:56.644074 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:56.644046 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 11:58:56.644236 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:56.644146 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs podName:76443aa4-c20f-4f32-8bbd-f88a776b0b34 nodeName:}" failed. No retries permitted until 2026-07-15 11:58:58.644124473 +0000 UTC m=+6.057652805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs") pod "network-metrics-daemon-l4d6j" (UID: "76443aa4-c20f-4f32-8bbd-f88a776b0b34") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 11:58:56.644750 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:56.644727 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 11:58:56.644881 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:56.644797 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret podName:977f5c84-a6ea-4927-93ad-9dd0e23853a8 nodeName:}" failed. No retries permitted until 2026-07-15 11:58:58.644772124 +0000 UTC m=+6.058300454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret") pod "global-pull-secret-syncer-ggb88" (UID: "977f5c84-a6ea-4927-93ad-9dd0e23853a8") : object "kube-system"/"original-pull-secret" not registered Jul 15 11:58:56.744434 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:56.744390 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rw8\" (UniqueName: \"kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8\") pod \"network-check-target-pd45b\" (UID: \"0c8aff4d-50ce-4f75-84fb-786fe725d46e\") " pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:58:56.744626 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:56.744574 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 11:58:56.744626 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:56.744598 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 11:58:56.744626 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:56.744612 2570 projected.go:196] Error preparing data for projected volume kube-api-access-f8rw8 for pod openshift-network-diagnostics/network-check-target-pd45b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 11:58:56.744814 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:56.744687 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8 podName:0c8aff4d-50ce-4f75-84fb-786fe725d46e nodeName:}" failed. No retries permitted until 2026-07-15 11:58:58.744653009 +0000 UTC m=+6.158181359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8rw8" (UniqueName: "kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8") pod "network-check-target-pd45b" (UID: "0c8aff4d-50ce-4f75-84fb-786fe725d46e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 11:58:57.060103 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:57.059905 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 11:53:55 +0000 UTC" deadline="2028-01-29 01:31:24.168427283 +0000 UTC" Jul 15 11:58:57.060103 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:57.060048 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13501h32m27.108385754s" Jul 15 11:58:57.089753 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:57.088829 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:58:57.089753 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:57.088966 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:58:57.089753 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:57.089374 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:58:57.089753 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:57.089467 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:58:57.089753 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:57.089552 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:58:57.089753 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:57.089645 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:58:58.662870 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:58.662832 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:58:58.663342 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:58.662936 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs\") pod \"network-metrics-daemon-l4d6j\" (UID: \"76443aa4-c20f-4f32-8bbd-f88a776b0b34\") " pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:58:58.663342 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:58.663066 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 11:58:58.663342 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:58.663130 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs podName:76443aa4-c20f-4f32-8bbd-f88a776b0b34 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:02.663110442 +0000 UTC m=+10.076638774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs") pod "network-metrics-daemon-l4d6j" (UID: "76443aa4-c20f-4f32-8bbd-f88a776b0b34") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 11:58:58.663571 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:58.663516 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 11:58:58.663571 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:58.663567 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret podName:977f5c84-a6ea-4927-93ad-9dd0e23853a8 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:02.663552331 +0000 UTC m=+10.077080662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret") pod "global-pull-secret-syncer-ggb88" (UID: "977f5c84-a6ea-4927-93ad-9dd0e23853a8") : object "kube-system"/"original-pull-secret" not registered Jul 15 11:58:58.764057 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:58.764019 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rw8\" (UniqueName: \"kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8\") pod \"network-check-target-pd45b\" (UID: \"0c8aff4d-50ce-4f75-84fb-786fe725d46e\") " pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:58:58.764241 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:58.764228 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 11:58:58.764303 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:58.764246 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 11:58:58.764303 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:58.764258 2570 projected.go:196] Error preparing data for projected volume kube-api-access-f8rw8 for pod openshift-network-diagnostics/network-check-target-pd45b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 11:58:58.764400 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:58.764312 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8 podName:0c8aff4d-50ce-4f75-84fb-786fe725d46e nodeName:}" failed. No retries permitted until 2026-07-15 11:59:02.764293959 +0000 UTC m=+10.177822303 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8rw8" (UniqueName: "kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8") pod "network-check-target-pd45b" (UID: "0c8aff4d-50ce-4f75-84fb-786fe725d46e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 11:58:59.088596 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:59.088521 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:58:59.088765 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:59.088647 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:58:59.089009 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:59.088989 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:58:59.089100 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:59.089067 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:58:59.089100 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:58:59.089098 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:58:59.089187 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:58:59.089138 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:01.088571 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:01.088534 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:01.089060 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:01.088674 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:01.089128 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:01.089064 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:01.089184 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:01.089156 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:01.089242 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:01.089209 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:01.089306 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:01.089288 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:02.697410 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:02.697371 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:02.697907 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:02.697466 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs\") pod \"network-metrics-daemon-l4d6j\" (UID: \"76443aa4-c20f-4f32-8bbd-f88a776b0b34\") " pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:02.697907 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:02.697595 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 11:59:02.697907 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:02.697644 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs podName:76443aa4-c20f-4f32-8bbd-f88a776b0b34 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:10.697630825 +0000 UTC m=+18.111159154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs") pod "network-metrics-daemon-l4d6j" (UID: "76443aa4-c20f-4f32-8bbd-f88a776b0b34") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 11:59:02.698072 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:02.698050 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 11:59:02.698125 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:02.698099 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret podName:977f5c84-a6ea-4927-93ad-9dd0e23853a8 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:10.698079998 +0000 UTC m=+18.111608336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret") pod "global-pull-secret-syncer-ggb88" (UID: "977f5c84-a6ea-4927-93ad-9dd0e23853a8") : object "kube-system"/"original-pull-secret" not registered Jul 15 11:59:02.797961 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:02.797919 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rw8\" (UniqueName: \"kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8\") pod \"network-check-target-pd45b\" (UID: \"0c8aff4d-50ce-4f75-84fb-786fe725d46e\") " pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:02.798142 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:02.798085 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 11:59:02.798142 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:02.798129 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 11:59:02.798142 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:02.798140 2570 projected.go:196] Error preparing data for projected volume kube-api-access-f8rw8 for pod openshift-network-diagnostics/network-check-target-pd45b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 11:59:02.798291 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:02.798199 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8 podName:0c8aff4d-50ce-4f75-84fb-786fe725d46e nodeName:}" failed. No retries permitted until 2026-07-15 11:59:10.798181415 +0000 UTC m=+18.211709746 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8rw8" (UniqueName: "kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8") pod "network-check-target-pd45b" (UID: "0c8aff4d-50ce-4f75-84fb-786fe725d46e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 11:59:03.086653 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:03.086580 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:03.086903 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:03.086710 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:03.087135 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:03.087117 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:03.087229 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:03.087210 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:03.087295 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:03.087283 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:03.087373 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:03.087354 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:05.086869 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:05.086838 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:05.087261 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:05.086839 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:05.087261 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:05.086941 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:05.087261 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:05.086959 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:05.087261 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:05.087088 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:05.087261 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:05.087232 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:07.086428 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:07.086234 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:07.086820 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:07.086241 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:07.086820 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:07.086291 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:07.086820 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:07.086522 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:07.086820 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:07.086607 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:07.086820 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:07.086704 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:09.086751 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:09.086711 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:09.086751 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:09.086733 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:09.087258 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:09.086728 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:09.087258 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:09.086860 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:09.087258 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:09.086963 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:09.087258 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:09.087051 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:10.757974 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:10.757942 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:10.758501 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:10.758030 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs\") pod \"network-metrics-daemon-l4d6j\" (UID: \"76443aa4-c20f-4f32-8bbd-f88a776b0b34\") " pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:10.758501 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:10.758126 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 11:59:10.758501 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:10.758131 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 11:59:10.758501 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:10.758189 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs podName:76443aa4-c20f-4f32-8bbd-f88a776b0b34 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:26.758169486 +0000 UTC m=+34.171697863 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs") pod "network-metrics-daemon-l4d6j" (UID: "76443aa4-c20f-4f32-8bbd-f88a776b0b34") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 11:59:10.758501 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:10.758203 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret podName:977f5c84-a6ea-4927-93ad-9dd0e23853a8 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:26.758197145 +0000 UTC m=+34.171725474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret") pod "global-pull-secret-syncer-ggb88" (UID: "977f5c84-a6ea-4927-93ad-9dd0e23853a8") : object "kube-system"/"original-pull-secret" not registered Jul 15 11:59:10.859392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:10.859349 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rw8\" (UniqueName: \"kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8\") pod \"network-check-target-pd45b\" (UID: \"0c8aff4d-50ce-4f75-84fb-786fe725d46e\") " pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:10.859585 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:10.859524 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 11:59:10.859585 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:10.859552 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 11:59:10.859585 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:10.859567 2570 projected.go:196] Error preparing data for projected volume kube-api-access-f8rw8 for pod openshift-network-diagnostics/network-check-target-pd45b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 11:59:10.859764 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:10.859627 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8 podName:0c8aff4d-50ce-4f75-84fb-786fe725d46e nodeName:}" failed. No retries permitted until 2026-07-15 11:59:26.859607154 +0000 UTC m=+34.273135483 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8rw8" (UniqueName: "kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8") pod "network-check-target-pd45b" (UID: "0c8aff4d-50ce-4f75-84fb-786fe725d46e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 11:59:11.088627 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:11.088554 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:11.088627 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:11.088591 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:11.088809 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:11.088670 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:11.088809 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:11.088694 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:11.088809 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:11.088728 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:11.088919 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:11.088833 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:13.087192 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:13.087158 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:13.087614 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:13.087282 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:13.087701 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:13.087675 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:13.087853 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:13.087836 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:13.087928 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:13.087866 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:13.087979 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:13.087946 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:15.086024 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:15.085989 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:15.086024 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:15.086014 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:15.086590 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:15.086124 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:15.086590 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:15.086132 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:15.086590 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:15.086229 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:15.086590 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:15.086302 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:16.154336 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:16.154297 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" event={"ID":"e60c06f6-afda-41ff-b0af-eb7a11e513a3","Type":"ContainerStarted","Data":"4246492a911a2ca6a81ebdc2bc1067d0d481e9a3058a4c6f662d3eff381ddd3d"} Jul 15 11:59:16.156159 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:16.156138 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" event={"ID":"52d7772a-574c-4960-9541-e138399719f3","Type":"ContainerStarted","Data":"45e7c0295347503bce83f33a653a9a65a45ac14ea3683c2fe74e1c97137a9933"} Jul 15 11:59:16.156247 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:16.156213 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" event={"ID":"52d7772a-574c-4960-9541-e138399719f3","Type":"ContainerStarted","Data":"9d442bcca7520593825f327b238d1bad854f0814a6aafb5c9d6127112926ef69"} Jul 15 11:59:16.157671 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:16.157646 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nzwbz" event={"ID":"ac9ea448-c0c5-4782-833e-e767a709de84","Type":"ContainerStarted","Data":"cd189774227d48db78bc79e4514c5695d92c303047e451283b8ad61e3bb925b0"} Jul 15 11:59:16.160081 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:16.160024 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-53.ec2.internal" event={"ID":"72a235d1a8bfb42e6972f74d75275638","Type":"ContainerStarted","Data":"5ff64794d4f2373225f90050f7767a80c6553f2742600083e10a644ca4ad5d3e"} Jul 15 11:59:16.172299 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:16.172257 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dzq6d" podStartSLOduration=2.821142793 podStartE2EDuration="23.172244491s" totalImagesPullingTime="20.351101698s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:59:16.172035889 +0000 UTC m=+23.585564259" watchObservedRunningTime="2026-07-15 11:59:16.172244491 +0000 UTC m=+23.585772842" Jul 15 11:59:16.202119 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:16.202078 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nzwbz" podStartSLOduration=2.832816473 podStartE2EDuration="23.202063784s" totalImagesPullingTime="20.369247311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:59:16.201929596 +0000 UTC m=+23.615457960" watchObservedRunningTime="2026-07-15 11:59:16.202063784 +0000 UTC m=+23.615592134" Jul 15 11:59:16.215418 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:16.215376 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-53.ec2.internal" podStartSLOduration=22.215363517 podStartE2EDuration="22.215363517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:59:16.215058712 +0000 UTC m=+23.628587065" watchObservedRunningTime="2026-07-15 11:59:16.215363517 +0000 UTC m=+23.628891867" Jul 15 11:59:17.086832 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.086430 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:17.086950 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.086430 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:17.086950 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:17.086929 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:17.087074 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:17.086970 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:17.087074 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.086509 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:17.087074 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:17.087040 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:17.163613 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.163575 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-snrv8" event={"ID":"e507f9d6-cd28-4d22-9dfe-5869963a8c48","Type":"ContainerStarted","Data":"c8e6d9216d5fcfcd790b88e5607a56952ac4da1664f181ca849d42ad35738e6c"} Jul 15 11:59:17.165003 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.164979 2570 generic.go:356] "Generic (PLEG): container finished" podID="13a16a94-e9e0-4c90-9490-694591b8d873" containerID="186f9af1e752dc54ac7336babe03279b24eb6a41bf7da1c8751d53083dd1cd8e" exitCode=0 Jul 15 11:59:17.165087 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.165066 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtb" event={"ID":"13a16a94-e9e0-4c90-9490-694591b8d873","Type":"ContainerDied","Data":"186f9af1e752dc54ac7336babe03279b24eb6a41bf7da1c8751d53083dd1cd8e"} Jul 15 11:59:17.166470 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.166360 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9zcb8" event={"ID":"3a23f0bc-7022-4e72-b3cf-0a63b6e681f8","Type":"ContainerStarted","Data":"6de29a21dce3743b15bf9d710466f91f5046b36b8def064b18667385dd2e1292"} Jul 15 11:59:17.167978 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.167958 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ckhmn" event={"ID":"86361831-462f-4f0d-b679-adefb3cd226c","Type":"ContainerStarted","Data":"6bf9d83a9febeb3d4a8923561c1fe283c453b036211fb72971ff5777e3bb8cae"} Jul 15 11:59:17.169369 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.169349 2570 generic.go:356] "Generic (PLEG): container finished" podID="21e4333f5fd0a8a3d223f0d7258f6334" containerID="ebebd8416ecdce8982166ff9e8ebc3889d8d48e8e302c0e65cb8668c756b0ffd" exitCode=0 Jul 15 11:59:17.169452 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.169409 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal" event={"ID":"21e4333f5fd0a8a3d223f0d7258f6334","Type":"ContainerDied","Data":"ebebd8416ecdce8982166ff9e8ebc3889d8d48e8e302c0e65cb8668c756b0ffd"} Jul 15 11:59:17.170789 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.170764 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fr6q8" event={"ID":"88807a13-409e-4b95-b399-43ed30674850","Type":"ContainerStarted","Data":"ce05a4959fac75fef5f709ae1253b1310c052a7b7c0df813009ff84d420213b2"} Jul 15 11:59:17.172179 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.172160 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" event={"ID":"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358","Type":"ContainerStarted","Data":"2970bd7ea24dd15a3b19c12bdb7ce1e3a37fc660e2a741dd9ef48837ce722a60"} Jul 15 11:59:17.174669 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.174651 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 11:59:17.174966 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.174949 2570 generic.go:356] "Generic (PLEG): container finished" podID="52d7772a-574c-4960-9541-e138399719f3" containerID="45e7c0295347503bce83f33a653a9a65a45ac14ea3683c2fe74e1c97137a9933" exitCode=1 Jul 15 11:59:17.175046 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.175029 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" event={"ID":"52d7772a-574c-4960-9541-e138399719f3","Type":"ContainerDied","Data":"45e7c0295347503bce83f33a653a9a65a45ac14ea3683c2fe74e1c97137a9933"} Jul 15 11:59:17.175118 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.175105 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" event={"ID":"52d7772a-574c-4960-9541-e138399719f3","Type":"ContainerStarted","Data":"4e65839367bb12ebe92037fdd78ba70182d8ec2974a941b0b57c19107b441474"} Jul 15 11:59:17.175161 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.175150 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" event={"ID":"52d7772a-574c-4960-9541-e138399719f3","Type":"ContainerStarted","Data":"b224615d1654bf326fe047224efbccc5fc72c0851c58acab57e10a5160e3247c"} Jul 15 11:59:17.175209 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.175193 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" event={"ID":"52d7772a-574c-4960-9541-e138399719f3","Type":"ContainerStarted","Data":"4a7f25ebbe1a5ff6788171db3ca8eeec8b0c1c3e92d4cbbc0906cd20481560ce"} Jul 15 11:59:17.175247 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.175232 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" event={"ID":"52d7772a-574c-4960-9541-e138399719f3","Type":"ContainerStarted","Data":"a141368f7cd42b23516aad35c5361c5b4353e67d255f913347c60a6ad4465633"} Jul 15 11:59:17.181771 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.181719 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-snrv8" podStartSLOduration=3.791011893 podStartE2EDuration="24.18170932s" totalImagesPullingTime="20.390697427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:59:17.181280383 +0000 UTC m=+24.594808734" watchObservedRunningTime="2026-07-15 11:59:17.18170932 +0000 UTC m=+24.595237670" Jul 15 11:59:17.226584 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.226512 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ckhmn" podStartSLOduration=3.9483805309999997 podStartE2EDuration="24.226490952s" totalImagesPullingTime="20.278110421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:59:17.226093494 +0000 UTC m=+24.639621845" watchObservedRunningTime="2026-07-15 11:59:17.226490952 +0000 UTC m=+24.640019305" Jul 15 11:59:17.238684 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.238631 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fr6q8" podStartSLOduration=3.978726306 podStartE2EDuration="24.238610031s" totalImagesPullingTime="20.259883725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:59:17.2385833 +0000 UTC m=+24.652111652" watchObservedRunningTime="2026-07-15 11:59:17.238610031 +0000 UTC m=+24.652138384" Jul 15 11:59:17.251896 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.251847 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9zcb8" podStartSLOduration=3.9089232640000002 podStartE2EDuration="24.251827072s" totalImagesPullingTime="20.342903808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:59:17.251816624 +0000 UTC m=+24.665344994" watchObservedRunningTime="2026-07-15 11:59:17.251827072 +0000 UTC m=+24.665355427" Jul 15 11:59:17.795020 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:17.794999 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 11:59:18.089381 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:18.089211 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T11:59:17.795016665Z","UUID":"93a98159-c1f8-4e6f-9345-6920133e24f3","Handler":null,"Name":"","Endpoint":""} Jul 15 11:59:18.090913 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:18.090892 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 11:59:18.091097 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:18.090934 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 11:59:18.179403 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:18.179362 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal" event={"ID":"21e4333f5fd0a8a3d223f0d7258f6334","Type":"ContainerStarted","Data":"5d0527f33d06540be7044a82f8471f1189e99223700f83cb951890f7bfc1f531"} Jul 15 11:59:18.181540 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:18.181508 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" event={"ID":"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358","Type":"ContainerStarted","Data":"3ead8c1911e9d4c3b60ad6256d90008b732da71232eed42261fe0004e38b2c00"} Jul 15 11:59:19.086890 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:19.086862 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:19.087056 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:19.086911 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:19.087056 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:19.086864 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:19.087056 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:19.086977 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:19.087056 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:19.087039 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:19.087202 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:19.087119 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:19.186495 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:19.186450 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" event={"ID":"cc7e18cf-ef4d-4dd2-8d45-efd745c4b358","Type":"ContainerStarted","Data":"0dfbbb29596c11729f09450b7107ac8153249848797d2bfc84296b5b9bee5f8a"} Jul 15 11:59:19.189756 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:19.189711 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 11:59:19.190373 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:19.190346 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" event={"ID":"52d7772a-574c-4960-9541-e138399719f3","Type":"ContainerStarted","Data":"657749cc2c6d0cb2526a9a47d8a0fe8a09887a7d665ada37a90f02d7d0499ce2"} Jul 15 11:59:19.213325 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:19.213270 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q2lp2" podStartSLOduration=11.929609125 podStartE2EDuration="26.213252828s" totalImagesPullingTime="14.283643703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:58:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:59:19.213029164 +0000 UTC m=+26.626557515" watchObservedRunningTime="2026-07-15 11:59:19.213252828 +0000 UTC m=+26.626781178" Jul 15 11:59:19.213521 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:19.213420 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-53.ec2.internal" podStartSLOduration=25.21341177 podStartE2EDuration="25.21341177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:59:18.243509134 +0000 UTC m=+25.657037485" watchObservedRunningTime="2026-07-15 11:59:19.21341177 +0000 UTC m=+26.626940125" Jul 15 11:59:20.741163 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:20.741128 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fr6q8" Jul 15 11:59:20.741799 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:20.741781 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fr6q8" Jul 15 11:59:21.086727 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:21.086650 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:21.086910 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:21.086664 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:21.086910 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:21.086784 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:21.086910 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:21.086667 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:21.086910 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:21.086868 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:21.087138 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:21.086955 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:21.195016 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:21.194992 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fr6q8" Jul 15 11:59:21.195416 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:21.195399 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fr6q8" Jul 15 11:59:22.199324 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:22.199128 2570 generic.go:356] "Generic (PLEG): container finished" podID="13a16a94-e9e0-4c90-9490-694591b8d873" containerID="37372f0dd64e1484f8c942974a67459b48fefa11bf5499657a27c4975c136166" exitCode=0 Jul 15 11:59:22.200173 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:22.199208 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtb" event={"ID":"13a16a94-e9e0-4c90-9490-694591b8d873","Type":"ContainerDied","Data":"37372f0dd64e1484f8c942974a67459b48fefa11bf5499657a27c4975c136166"} Jul 15 11:59:22.203324 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:22.203305 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 11:59:22.203686 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:22.203664 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" event={"ID":"52d7772a-574c-4960-9541-e138399719f3","Type":"ContainerStarted","Data":"62ea6352906cdb1e39b03c394a265702fc2c3fa4a13b7eaa369ca51c0d5bb8c9"} Jul 15 11:59:22.204195 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:22.204179 2570 scope.go:117] "RemoveContainer" containerID="45e7c0295347503bce83f33a653a9a65a45ac14ea3683c2fe74e1c97137a9933" Jul 15 11:59:23.087100 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.086874 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:23.087392 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:23.087192 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:23.087392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.086996 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:23.087392 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.086957 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:23.087392 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:23.087294 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:23.087616 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:23.087389 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:23.211318 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.211294 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 11:59:23.211696 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.211672 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" event={"ID":"52d7772a-574c-4960-9541-e138399719f3","Type":"ContainerStarted","Data":"69ea692180534f75da0c65f8c068da8a532992df8ec519e48e84e760a728b266"} Jul 15 11:59:23.214399 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.214351 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:59:23.214549 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.214429 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:59:23.214549 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.214478 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:59:23.215199 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.215167 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ggb88"] Jul 15 11:59:23.215389 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.215232 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:23.215389 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:23.215334 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:23.216033 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.216011 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pd45b"] Jul 15 11:59:23.216109 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.216080 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:23.216174 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:23.216155 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:23.234335 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.234252 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:59:23.234583 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.234560 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:59:23.235205 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.235159 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l4d6j"] Jul 15 11:59:23.235315 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.235273 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:23.235383 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:23.235367 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:23.242727 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:23.242671 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" podStartSLOduration=9.653226466 podStartE2EDuration="30.242654297s" totalImagesPullingTime="20.589427831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:59:23.24224733 +0000 UTC m=+30.655775705" watchObservedRunningTime="2026-07-15 11:59:23.242654297 +0000 UTC m=+30.656182635" Jul 15 11:59:24.215508 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:24.215477 2570 generic.go:356] "Generic (PLEG): container finished" podID="13a16a94-e9e0-4c90-9490-694591b8d873" containerID="f9c1b308c5d625259c7e68b51489429193c39042a327cfda060718567ecb872b" exitCode=0 Jul 15 11:59:24.216066 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:24.215555 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtb" event={"ID":"13a16a94-e9e0-4c90-9490-694591b8d873","Type":"ContainerDied","Data":"f9c1b308c5d625259c7e68b51489429193c39042a327cfda060718567ecb872b"} Jul 15 11:59:25.086548 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:25.086457 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:25.086548 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:25.086468 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:25.086848 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:25.086641 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:25.086848 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:25.086673 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:25.086848 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:25.086800 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:25.087001 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:25.086858 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:26.222644 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:26.222599 2570 generic.go:356] "Generic (PLEG): container finished" podID="13a16a94-e9e0-4c90-9490-694591b8d873" containerID="e802f3c0281b3aa7e4094ced63cf34ab20ea3b72658fdcb92d757ead243409c5" exitCode=0 Jul 15 11:59:26.223051 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:26.222668 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtb" event={"ID":"13a16a94-e9e0-4c90-9490-694591b8d873","Type":"ContainerDied","Data":"e802f3c0281b3aa7e4094ced63cf34ab20ea3b72658fdcb92d757ead243409c5"} Jul 15 11:59:26.775549 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:26.775340 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs\") pod \"network-metrics-daemon-l4d6j\" (UID: \"76443aa4-c20f-4f32-8bbd-f88a776b0b34\") " pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:26.775702 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:26.775579 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:26.775702 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:26.775491 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 11:59:26.775702 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:26.775667 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 11:59:26.775702 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:26.775675 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs podName:76443aa4-c20f-4f32-8bbd-f88a776b0b34 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:58.775657609 +0000 UTC m=+66.189185938 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs") pod "network-metrics-daemon-l4d6j" (UID: "76443aa4-c20f-4f32-8bbd-f88a776b0b34") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 11:59:26.775702 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:26.775703 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret podName:977f5c84-a6ea-4927-93ad-9dd0e23853a8 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:58.775692582 +0000 UTC m=+66.189220910 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret") pod "global-pull-secret-syncer-ggb88" (UID: "977f5c84-a6ea-4927-93ad-9dd0e23853a8") : object "kube-system"/"original-pull-secret" not registered Jul 15 11:59:26.876402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:26.876364 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rw8\" (UniqueName: \"kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8\") pod \"network-check-target-pd45b\" (UID: \"0c8aff4d-50ce-4f75-84fb-786fe725d46e\") " pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:26.876566 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:26.876517 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 11:59:26.876566 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:26.876538 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 11:59:26.876566 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:26.876548 2570 projected.go:196] Error preparing data for projected volume kube-api-access-f8rw8 for pod openshift-network-diagnostics/network-check-target-pd45b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 11:59:26.876677 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:26.876602 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8 podName:0c8aff4d-50ce-4f75-84fb-786fe725d46e nodeName:}" failed. No retries permitted until 2026-07-15 11:59:58.876585743 +0000 UTC m=+66.290114073 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8rw8" (UniqueName: "kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8") pod "network-check-target-pd45b" (UID: "0c8aff4d-50ce-4f75-84fb-786fe725d46e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 11:59:27.085937 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:27.085851 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:27.086094 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:27.085862 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:27.086169 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:27.086144 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:27.086222 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:27.086002 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:27.086276 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:27.086222 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:27.086320 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:27.086291 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:27.649500 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:27.649463 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:27.755449 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:27.755414 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:27.871858 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:27.871824 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:28.567788 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:28.567759 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:28.668140 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:28.668106 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:28.844415 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:28.844332 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:28.946295 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:28.946255 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:29.085968 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.085936 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:29.086145 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.085938 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:29.086145 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:29.086063 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ggb88" podUID="977f5c84-a6ea-4927-93ad-9dd0e23853a8" Jul 15 11:59:29.086145 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.085944 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:29.086306 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:29.086138 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pd45b" podUID="0c8aff4d-50ce-4f75-84fb-786fe725d46e" Jul 15 11:59:29.086306 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:29.086195 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l4d6j" podUID="76443aa4-c20f-4f32-8bbd-f88a776b0b34" Jul 15 11:59:29.376016 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.375979 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-53.ec2.internal" event="NodeReady" Jul 15 11:59:29.376182 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.376146 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 11:59:29.428178 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.428145 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-24s4z"] Jul 15 11:59:29.430837 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.430804 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5zvqc"] Jul 15 11:59:29.430964 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.430922 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:29.432651 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.432629 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bqsqx\"" Jul 15 11:59:29.432798 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.432780 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5zvqc" Jul 15 11:59:29.433776 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.433753 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 11:59:29.434336 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.434297 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 11:59:29.434598 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.434579 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 11:59:29.434667 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.434599 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 11:59:29.436262 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.436244 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 11:59:29.436356 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.436321 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zh86n\"" Jul 15 11:59:29.447858 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.447838 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5zvqc"] Jul 15 11:59:29.447980 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.447888 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-24s4z"] Jul 15 11:59:29.595682 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.595646 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a92dead1-e314-46f5-9a12-c34efd573904-tmp-dir\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:29.595874 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.595712 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b959ddc-0f8a-473d-a311-afcfa283fed3-cert\") pod \"ingress-canary-5zvqc\" (UID: \"8b959ddc-0f8a-473d-a311-afcfa283fed3\") " pod="openshift-ingress-canary/ingress-canary-5zvqc" Jul 15 11:59:29.595874 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.595758 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzxg4\" (UniqueName: \"kubernetes.io/projected/8b959ddc-0f8a-473d-a311-afcfa283fed3-kube-api-access-rzxg4\") pod \"ingress-canary-5zvqc\" (UID: \"8b959ddc-0f8a-473d-a311-afcfa283fed3\") " pod="openshift-ingress-canary/ingress-canary-5zvqc" Jul 15 11:59:29.595874 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.595834 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a92dead1-e314-46f5-9a12-c34efd573904-config-volume\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:29.596017 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.595892 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkqxr\" (UniqueName: \"kubernetes.io/projected/a92dead1-e314-46f5-9a12-c34efd573904-kube-api-access-vkqxr\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:29.596017 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.595950 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:29.696987 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.696950 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a92dead1-e314-46f5-9a12-c34efd573904-config-volume\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:29.696987 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.696992 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkqxr\" (UniqueName: \"kubernetes.io/projected/a92dead1-e314-46f5-9a12-c34efd573904-kube-api-access-vkqxr\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:29.697504 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.697035 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:29.697504 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.697072 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a92dead1-e314-46f5-9a12-c34efd573904-tmp-dir\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:29.697504 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.697099 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b959ddc-0f8a-473d-a311-afcfa283fed3-cert\") pod \"ingress-canary-5zvqc\" (UID: \"8b959ddc-0f8a-473d-a311-afcfa283fed3\") " pod="openshift-ingress-canary/ingress-canary-5zvqc" Jul 15 11:59:29.697504 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.697120 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzxg4\" (UniqueName: \"kubernetes.io/projected/8b959ddc-0f8a-473d-a311-afcfa283fed3-kube-api-access-rzxg4\") pod \"ingress-canary-5zvqc\" (UID: \"8b959ddc-0f8a-473d-a311-afcfa283fed3\") " pod="openshift-ingress-canary/ingress-canary-5zvqc" Jul 15 11:59:29.697504 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:29.697203 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 11:59:29.697504 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:29.697269 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls podName:a92dead1-e314-46f5-9a12-c34efd573904 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:30.197252644 +0000 UTC m=+37.610780992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls") pod "dns-default-24s4z" (UID: "a92dead1-e314-46f5-9a12-c34efd573904") : secret "dns-default-metrics-tls" not found Jul 15 11:59:29.697504 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.697419 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a92dead1-e314-46f5-9a12-c34efd573904-tmp-dir\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:29.697726 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.697584 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a92dead1-e314-46f5-9a12-c34efd573904-config-volume\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:29.702049 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.702011 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b959ddc-0f8a-473d-a311-afcfa283fed3-cert\") pod \"ingress-canary-5zvqc\" (UID: \"8b959ddc-0f8a-473d-a311-afcfa283fed3\") " pod="openshift-ingress-canary/ingress-canary-5zvqc" Jul 15 11:59:29.705228 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.705207 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzxg4\" (UniqueName: \"kubernetes.io/projected/8b959ddc-0f8a-473d-a311-afcfa283fed3-kube-api-access-rzxg4\") pod \"ingress-canary-5zvqc\" (UID: \"8b959ddc-0f8a-473d-a311-afcfa283fed3\") " pod="openshift-ingress-canary/ingress-canary-5zvqc" Jul 15 11:59:29.705348 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.705243 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkqxr\" (UniqueName: \"kubernetes.io/projected/a92dead1-e314-46f5-9a12-c34efd573904-kube-api-access-vkqxr\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:29.755453 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:29.755424 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5zvqc" Jul 15 11:59:30.202568 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:30.202537 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:30.202763 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:30.202678 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 11:59:30.202850 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:30.202765 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls podName:a92dead1-e314-46f5-9a12-c34efd573904 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:31.202723165 +0000 UTC m=+38.616251506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls") pod "dns-default-24s4z" (UID: "a92dead1-e314-46f5-9a12-c34efd573904") : secret "dns-default-metrics-tls" not found Jul 15 11:59:30.516095 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:30.515928 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5zvqc"] Jul 15 11:59:30.519341 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:59:30.519276 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b959ddc_0f8a_473d_a311_afcfa283fed3.slice/crio-10ac2e27ce2257d8239f10d9ade44cf8043ec59f2c2b922c7b78c724527de729 WatchSource:0}: Error finding container 10ac2e27ce2257d8239f10d9ade44cf8043ec59f2c2b922c7b78c724527de729: Status 404 returned error can't find the container with id 10ac2e27ce2257d8239f10d9ade44cf8043ec59f2c2b922c7b78c724527de729 Jul 15 11:59:31.086519 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:31.086487 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:31.087252 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:31.086487 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:31.087252 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:31.086487 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:31.088938 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:31.088916 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 11:59:31.089079 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:31.088947 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 11:59:31.089386 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:31.089367 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 11:59:31.089520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:31.089368 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f876h\"" Jul 15 11:59:31.089520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:31.089442 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7c5z8\"" Jul 15 11:59:31.089520 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:31.089377 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 11:59:31.210676 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:31.210636 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:31.210872 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:31.210842 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 11:59:31.210928 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:31.210921 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls podName:a92dead1-e314-46f5-9a12-c34efd573904 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:33.210897176 +0000 UTC m=+40.624425510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls") pod "dns-default-24s4z" (UID: "a92dead1-e314-46f5-9a12-c34efd573904") : secret "dns-default-metrics-tls" not found Jul 15 11:59:31.236714 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:31.236547 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5zvqc" event={"ID":"8b959ddc-0f8a-473d-a311-afcfa283fed3","Type":"ContainerStarted","Data":"10ac2e27ce2257d8239f10d9ade44cf8043ec59f2c2b922c7b78c724527de729"} Jul 15 11:59:31.239859 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:31.239832 2570 generic.go:356] "Generic (PLEG): container finished" podID="13a16a94-e9e0-4c90-9490-694591b8d873" containerID="c07b0bfee107c896eb7ce38b0b6eb638b52b9b0e0cadeff815a1449a275ff192" exitCode=0 Jul 15 11:59:31.239859 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:31.239869 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtb" event={"ID":"13a16a94-e9e0-4c90-9490-694591b8d873","Type":"ContainerDied","Data":"c07b0bfee107c896eb7ce38b0b6eb638b52b9b0e0cadeff815a1449a275ff192"} Jul 15 11:59:32.246136 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:32.246102 2570 generic.go:356] "Generic (PLEG): container finished" podID="13a16a94-e9e0-4c90-9490-694591b8d873" containerID="d7c57143d8f95c2ce3296fb46f0ef4f79368c8f13258e265888d0b5af7b185cc" exitCode=0 Jul 15 11:59:32.246551 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:32.246175 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtb" event={"ID":"13a16a94-e9e0-4c90-9490-694591b8d873","Type":"ContainerDied","Data":"d7c57143d8f95c2ce3296fb46f0ef4f79368c8f13258e265888d0b5af7b185cc"} Jul 15 11:59:33.227460 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:33.227426 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:33.227643 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:33.227620 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 11:59:33.227716 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:33.227705 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls podName:a92dead1-e314-46f5-9a12-c34efd573904 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:37.227684762 +0000 UTC m=+44.641213109 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls") pod "dns-default-24s4z" (UID: "a92dead1-e314-46f5-9a12-c34efd573904") : secret "dns-default-metrics-tls" not found Jul 15 11:59:33.251889 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:33.251733 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pbtb" event={"ID":"13a16a94-e9e0-4c90-9490-694591b8d873","Type":"ContainerStarted","Data":"b73aeeebd914620aed68d225ddbd7e4d4487a44d7d8dd98a432836886bc8e9f4"} Jul 15 11:59:33.278368 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:33.278320 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2pbtb" podStartSLOduration=9.622910203 podStartE2EDuration="40.278305366s" totalImagesPullingTime="30.655395163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:58:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:59:33.277278694 +0000 UTC m=+40.690807045" watchObservedRunningTime="2026-07-15 11:59:33.278305366 +0000 UTC m=+40.691833716" Jul 15 11:59:33.512367 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:33.512269 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:33.601442 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:33.601408 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:34.255866 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:34.255830 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5zvqc" event={"ID":"8b959ddc-0f8a-473d-a311-afcfa283fed3","Type":"ContainerStarted","Data":"2347cc504bae61a5def28c048f5a53561ea7a2f42a0cb16ad1604d68798b67d1"} Jul 15 11:59:34.273903 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:34.273852 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5zvqc" podStartSLOduration=2.60780953 podStartE2EDuration="5.2738308s" totalImagesPullingTime="2.66602127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:59:34.272958347 +0000 UTC m=+41.686486697" watchObservedRunningTime="2026-07-15 11:59:34.2738308 +0000 UTC m=+41.687359150" Jul 15 11:59:34.312430 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:34.312396 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:34.473519 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:34.473471 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:35.957721 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:35.957692 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:37.253674 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:37.253635 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:37.254072 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:37.253805 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 11:59:37.254072 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:37.253870 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls podName:a92dead1-e314-46f5-9a12-c34efd573904 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:45.253852149 +0000 UTC m=+52.667380477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls") pod "dns-default-24s4z" (UID: "a92dead1-e314-46f5-9a12-c34efd573904") : secret "dns-default-metrics-tls" not found Jul 15 11:59:45.310963 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:45.310929 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 11:59:45.311471 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:45.311030 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 11:59:45.311471 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:45.311086 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls podName:a92dead1-e314-46f5-9a12-c34efd573904 nodeName:}" failed. No retries permitted until 2026-07-15 12:00:01.311071896 +0000 UTC m=+68.724600224 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls") pod "dns-default-24s4z" (UID: "a92dead1-e314-46f5-9a12-c34efd573904") : secret "dns-default-metrics-tls" not found Jul 15 11:59:49.549547 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:49.549507 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:49.625980 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:49.625954 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:50.751402 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.751363 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb"] Jul 15 11:59:50.753613 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.753596 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" Jul 15 11:59:50.758267 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.758244 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 15 11:59:50.758728 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.758709 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 15 11:59:50.758852 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.758712 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mppd2\"" Jul 15 11:59:50.758938 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.758920 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 15 11:59:50.758985 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.758937 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 15 11:59:50.781241 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.781210 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb"] Jul 15 11:59:50.851848 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.851814 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv9vn\" (UniqueName: \"kubernetes.io/projected/74bcecd1-b162-4ce3-b731-d47b223860a8-kube-api-access-kv9vn\") pod \"kube-storage-version-migrator-operator-799cb7f958-ftmvb\" (UID: \"74bcecd1-b162-4ce3-b731-d47b223860a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" Jul 15 11:59:50.852018 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.851854 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74bcecd1-b162-4ce3-b731-d47b223860a8-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-ftmvb\" (UID: \"74bcecd1-b162-4ce3-b731-d47b223860a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" Jul 15 11:59:50.852018 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.851963 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74bcecd1-b162-4ce3-b731-d47b223860a8-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-ftmvb\" (UID: \"74bcecd1-b162-4ce3-b731-d47b223860a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" Jul 15 11:59:50.858034 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.858007 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5c864b74c8-z6ccx"] Jul 15 11:59:50.859895 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.859880 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:50.867148 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.867116 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 15 11:59:50.867266 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.867249 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2q8rk\"" Jul 15 11:59:50.867328 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.867282 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 15 11:59:50.867328 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.867291 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 11:59:50.867601 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.867581 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 15 11:59:50.867601 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.867593 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 11:59:50.867755 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.867581 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 15 11:59:50.886758 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.886706 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c864b74c8-z6ccx"] Jul 15 11:59:50.952942 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.952902 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74bcecd1-b162-4ce3-b731-d47b223860a8-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-ftmvb\" (UID: \"74bcecd1-b162-4ce3-b731-d47b223860a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" Jul 15 11:59:50.953141 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.952969 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv9vn\" (UniqueName: \"kubernetes.io/projected/74bcecd1-b162-4ce3-b731-d47b223860a8-kube-api-access-kv9vn\") pod \"kube-storage-version-migrator-operator-799cb7f958-ftmvb\" (UID: \"74bcecd1-b162-4ce3-b731-d47b223860a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" Jul 15 11:59:50.953141 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.952999 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74bcecd1-b162-4ce3-b731-d47b223860a8-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-ftmvb\" (UID: \"74bcecd1-b162-4ce3-b731-d47b223860a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" Jul 15 11:59:50.953579 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.953550 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74bcecd1-b162-4ce3-b731-d47b223860a8-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-ftmvb\" (UID: \"74bcecd1-b162-4ce3-b731-d47b223860a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" Jul 15 11:59:50.957075 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.957056 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74bcecd1-b162-4ce3-b731-d47b223860a8-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-ftmvb\" (UID: \"74bcecd1-b162-4ce3-b731-d47b223860a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" Jul 15 11:59:50.960714 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:50.960692 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv9vn\" (UniqueName: \"kubernetes.io/projected/74bcecd1-b162-4ce3-b731-d47b223860a8-kube-api-access-kv9vn\") pod \"kube-storage-version-migrator-operator-799cb7f958-ftmvb\" (UID: \"74bcecd1-b162-4ce3-b731-d47b223860a8\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" Jul 15 11:59:51.054041 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.053951 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-stats-auth\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:51.054041 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.054005 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:51.054041 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.054022 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rff7s\" (UniqueName: \"kubernetes.io/projected/f8699bdf-f732-4921-a592-483bea1c84fb-kube-api-access-rff7s\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:51.057018 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.054445 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:51.057018 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.054535 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-default-certificate\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:51.062253 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.062234 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" Jul 15 11:59:51.092005 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.091974 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:51.155196 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.155165 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-stats-auth\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:51.155343 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.155235 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:51.155343 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.155256 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rff7s\" (UniqueName: \"kubernetes.io/projected/f8699bdf-f732-4921-a592-483bea1c84fb-kube-api-access-rff7s\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:51.155343 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.155302 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:51.155343 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.155321 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-default-certificate\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:51.155551 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:51.155463 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle podName:f8699bdf-f732-4921-a592-483bea1c84fb nodeName:}" failed. No retries permitted until 2026-07-15 11:59:51.655436626 +0000 UTC m=+59.068964963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle") pod "router-default-5c864b74c8-z6ccx" (UID: "f8699bdf-f732-4921-a592-483bea1c84fb") : configmap references non-existent config key: service-ca.crt Jul 15 11:59:51.155551 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:51.155493 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 11:59:51.155643 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:51.155571 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs podName:f8699bdf-f732-4921-a592-483bea1c84fb nodeName:}" failed. No retries permitted until 2026-07-15 11:59:51.655552112 +0000 UTC m=+59.069080444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs") pod "router-default-5c864b74c8-z6ccx" (UID: "f8699bdf-f732-4921-a592-483bea1c84fb") : secret "router-metrics-certs-default" not found Jul 15 11:59:51.158164 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.158139 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-stats-auth\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:51.158399 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.158383 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-default-certificate\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:51.163775 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.163732 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rff7s\" (UniqueName: \"kubernetes.io/projected/f8699bdf-f732-4921-a592-483bea1c84fb-kube-api-access-rff7s\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:51.186624 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.186594 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb"] Jul 15 11:59:51.189864 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:59:51.189837 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74bcecd1_b162_4ce3_b731_d47b223860a8.slice/crio-b4a5e2cf02142d586b4fec344b7dc3c5296b7fdfec470fb873ec1a3e6b5a5219 WatchSource:0}: Error finding container b4a5e2cf02142d586b4fec344b7dc3c5296b7fdfec470fb873ec1a3e6b5a5219: Status 404 returned error can't find the container with id b4a5e2cf02142d586b4fec344b7dc3c5296b7fdfec470fb873ec1a3e6b5a5219 Jul 15 11:59:51.191984 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.191958 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:51.296853 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.296812 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" event={"ID":"74bcecd1-b162-4ce3-b731-d47b223860a8","Type":"ContainerStarted","Data":"b4a5e2cf02142d586b4fec344b7dc3c5296b7fdfec470fb873ec1a3e6b5a5219"} Jul 15 11:59:51.659263 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.659229 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:51.659454 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:51.659300 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:51.659454 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:51.659411 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle podName:f8699bdf-f732-4921-a592-483bea1c84fb nodeName:}" failed. No retries permitted until 2026-07-15 11:59:52.659393707 +0000 UTC m=+60.072922035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle") pod "router-default-5c864b74c8-z6ccx" (UID: "f8699bdf-f732-4921-a592-483bea1c84fb") : configmap references non-existent config key: service-ca.crt Jul 15 11:59:51.659454 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:51.659414 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 11:59:51.659454 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:51.659451 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs podName:f8699bdf-f732-4921-a592-483bea1c84fb nodeName:}" failed. No retries permitted until 2026-07-15 11:59:52.659444244 +0000 UTC m=+60.072972573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs") pod "router-default-5c864b74c8-z6ccx" (UID: "f8699bdf-f732-4921-a592-483bea1c84fb") : secret "router-metrics-certs-default" not found Jul 15 11:59:52.667456 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:52.667412 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:52.667895 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:52.667525 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:52.667895 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:52.667588 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 11:59:52.667895 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:52.667652 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs podName:f8699bdf-f732-4921-a592-483bea1c84fb nodeName:}" failed. No retries permitted until 2026-07-15 11:59:54.667636796 +0000 UTC m=+62.081165125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs") pod "router-default-5c864b74c8-z6ccx" (UID: "f8699bdf-f732-4921-a592-483bea1c84fb") : secret "router-metrics-certs-default" not found Jul 15 11:59:52.667895 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:52.667667 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle podName:f8699bdf-f732-4921-a592-483bea1c84fb nodeName:}" failed. No retries permitted until 2026-07-15 11:59:54.667660075 +0000 UTC m=+62.081188404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle") pod "router-default-5c864b74c8-z6ccx" (UID: "f8699bdf-f732-4921-a592-483bea1c84fb") : configmap references non-existent config key: service-ca.crt Jul 15 11:59:53.302869 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:53.302834 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" event={"ID":"74bcecd1-b162-4ce3-b731-d47b223860a8","Type":"ContainerStarted","Data":"1c081c401ee4351d37cddbd59c88f144d1acadc4dd342456902beb2d207d0d06"} Jul 15 11:59:53.330715 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:53.330670 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" podStartSLOduration=1.344379682 podStartE2EDuration="3.330655141s" totalImagesPullingTime="1.986275459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 11:59:53.330028015 +0000 UTC m=+60.743556367" watchObservedRunningTime="2026-07-15 11:59:53.330655141 +0000 UTC m=+60.744183492" Jul 15 11:59:53.665759 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:53.665711 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:53.763375 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:53.763345 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:54.633141 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:54.633109 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:54.680335 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:54.680301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:54.680503 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:54.680377 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:54.680503 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:54.680450 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 11:59:54.680592 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:54.680506 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle podName:f8699bdf-f732-4921-a592-483bea1c84fb nodeName:}" failed. No retries permitted until 2026-07-15 11:59:58.680493351 +0000 UTC m=+66.094021680 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle") pod "router-default-5c864b74c8-z6ccx" (UID: "f8699bdf-f732-4921-a592-483bea1c84fb") : configmap references non-existent config key: service-ca.crt Jul 15 11:59:54.680592 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:54.680522 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs podName:f8699bdf-f732-4921-a592-483bea1c84fb nodeName:}" failed. No retries permitted until 2026-07-15 11:59:58.680515089 +0000 UTC m=+66.094043418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs") pod "router-default-5c864b74c8-z6ccx" (UID: "f8699bdf-f732-4921-a592-483bea1c84fb") : secret "router-metrics-certs-default" not found Jul 15 11:59:54.708043 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:54.708013 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:54.770927 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:54.770881 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:55.233087 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:55.233064 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7kxg9" Jul 15 11:59:55.396104 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:55.396070 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:55.507300 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:55.507225 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:55.651300 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:55.651262 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:55.752534 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:55.752499 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:56.200569 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:56.200540 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:56.299543 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:56.299513 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:56.398708 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:56.398675 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:57.044917 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.044881 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:57.146908 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.146880 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:57.290650 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.290618 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:57.397746 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.397713 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:57.407718 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.407694 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-289vh"] Jul 15 11:59:57.410238 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.410215 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:57.412188 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.412159 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 11:59:57.412290 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.412278 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 11:59:57.412718 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.412700 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7kwbt\"" Jul 15 11:59:57.413065 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.413045 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 11:59:57.413299 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.413284 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 11:59:57.424503 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.424482 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-289vh"] Jul 15 11:59:57.469203 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.469176 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:57.496499 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.496476 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjhdn\" (UniqueName: \"kubernetes.io/projected/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-api-access-bjhdn\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:57.496613 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.496519 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/144f5339-e458-46aa-acbe-f43a1e4dbec9-data-volume\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:57.496613 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.496570 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/144f5339-e458-46aa-acbe-f43a1e4dbec9-crio-socket\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:57.496613 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.496609 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:57.496714 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.496644 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:57.596964 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.596929 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/144f5339-e458-46aa-acbe-f43a1e4dbec9-data-volume\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:57.597135 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.596995 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/144f5339-e458-46aa-acbe-f43a1e4dbec9-crio-socket\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:57.597135 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.597056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:57.597135 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.597063 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/144f5339-e458-46aa-acbe-f43a1e4dbec9-crio-socket\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:57.597135 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.597093 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:57.597135 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.597123 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjhdn\" (UniqueName: \"kubernetes.io/projected/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-api-access-bjhdn\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:57.597386 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:57.597136 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 11:59:57.597386 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:57.597258 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 11:59:57.597386 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:57.597286 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm podName:144f5339-e458-46aa-acbe-f43a1e4dbec9 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:58.097265897 +0000 UTC m=+65.510794228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-289vh" (UID: "144f5339-e458-46aa-acbe-f43a1e4dbec9") : configmap "kube-rbac-proxy" not found Jul 15 11:59:57.597386 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:57.597324 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls podName:144f5339-e458-46aa-acbe-f43a1e4dbec9 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:58.097310678 +0000 UTC m=+65.510839008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-289vh" (UID: "144f5339-e458-46aa-acbe-f43a1e4dbec9") : secret "insights-runtime-extractor-tls" not found Jul 15 11:59:57.597386 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.597334 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/144f5339-e458-46aa-acbe-f43a1e4dbec9-data-volume\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:57.607895 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.607864 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjhdn\" (UniqueName: \"kubernetes.io/projected/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-api-access-bjhdn\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:57.756061 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.755980 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:57.926653 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:57.926615 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:58.102303 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.102272 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:58.102480 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.102401 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:58.102480 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:58.102412 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 11:59:58.102480 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:58.102473 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls podName:144f5339-e458-46aa-acbe-f43a1e4dbec9 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:59.102457629 +0000 UTC m=+66.515985984 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-289vh" (UID: "144f5339-e458-46aa-acbe-f43a1e4dbec9") : secret "insights-runtime-extractor-tls" not found Jul 15 11:59:58.102617 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:58.102514 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 11:59:58.102617 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:58.102575 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm podName:144f5339-e458-46aa-acbe-f43a1e4dbec9 nodeName:}" failed. No retries permitted until 2026-07-15 11:59:59.102561989 +0000 UTC m=+66.516090318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-289vh" (UID: "144f5339-e458-46aa-acbe-f43a1e4dbec9") : configmap "kube-rbac-proxy" not found Jul 15 11:59:58.265193 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.265148 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:58.335065 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.335028 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:58.338332 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.338309 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq"] Jul 15 11:59:58.340271 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.340256 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq" Jul 15 11:59:58.342326 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.342309 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 15 11:59:58.342868 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.342850 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 15 11:59:58.342948 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.342869 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-flcgv\"" Jul 15 11:59:58.343162 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.343130 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 15 11:59:58.343462 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.343444 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 15 11:59:58.352344 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.352287 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq"] Jul 15 11:59:58.405425 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.405390 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5z8q\" (UniqueName: \"kubernetes.io/projected/200a1940-87d2-4876-a7a3-06658dcd4ecb-kube-api-access-v5z8q\") pod \"managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq\" (UID: \"200a1940-87d2-4876-a7a3-06658dcd4ecb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq" Jul 15 11:59:58.405577 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.405480 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/200a1940-87d2-4876-a7a3-06658dcd4ecb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq\" (UID: \"200a1940-87d2-4876-a7a3-06658dcd4ecb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq" Jul 15 11:59:58.506633 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.506589 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/200a1940-87d2-4876-a7a3-06658dcd4ecb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq\" (UID: \"200a1940-87d2-4876-a7a3-06658dcd4ecb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq" Jul 15 11:59:58.506770 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.506697 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5z8q\" (UniqueName: \"kubernetes.io/projected/200a1940-87d2-4876-a7a3-06658dcd4ecb-kube-api-access-v5z8q\") pod \"managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq\" (UID: \"200a1940-87d2-4876-a7a3-06658dcd4ecb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq" Jul 15 11:59:58.509227 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.509205 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/200a1940-87d2-4876-a7a3-06658dcd4ecb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq\" (UID: \"200a1940-87d2-4876-a7a3-06658dcd4ecb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq" Jul 15 11:59:58.514724 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.514698 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5z8q\" (UniqueName: \"kubernetes.io/projected/200a1940-87d2-4876-a7a3-06658dcd4ecb-kube-api-access-v5z8q\") pod \"managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq\" (UID: \"200a1940-87d2-4876-a7a3-06658dcd4ecb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq" Jul 15 11:59:58.660260 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.660169 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq" Jul 15 11:59:58.709257 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.709222 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:58.709388 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.709336 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 11:59:58.709388 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:58.709372 2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 11:59:58.709513 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:58.709446 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs podName:f8699bdf-f732-4921-a592-483bea1c84fb nodeName:}" failed. No retries permitted until 2026-07-15 12:00:06.709429395 +0000 UTC m=+74.122957724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs") pod "router-default-5c864b74c8-z6ccx" (UID: "f8699bdf-f732-4921-a592-483bea1c84fb") : secret "router-metrics-certs-default" not found Jul 15 11:59:58.709580 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:58.709512 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle podName:f8699bdf-f732-4921-a592-483bea1c84fb nodeName:}" failed. No retries permitted until 2026-07-15 12:00:06.709489304 +0000 UTC m=+74.123017638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle") pod "router-default-5c864b74c8-z6ccx" (UID: "f8699bdf-f732-4921-a592-483bea1c84fb") : configmap references non-existent config key: service-ca.crt Jul 15 11:59:58.783003 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.782968 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq"] Jul 15 11:59:58.787613 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:59:58.787587 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod200a1940_87d2_4876_a7a3_06658dcd4ecb.slice/crio-faf567b9f042089afa0aabee3b92e9e9ec063c3862e15d2f6cebcddc2c95ce07 WatchSource:0}: Error finding container faf567b9f042089afa0aabee3b92e9e9ec063c3862e15d2f6cebcddc2c95ce07: Status 404 returned error can't find the container with id faf567b9f042089afa0aabee3b92e9e9ec063c3862e15d2f6cebcddc2c95ce07 Jul 15 11:59:58.810008 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.809982 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs\") pod \"network-metrics-daemon-l4d6j\" (UID: \"76443aa4-c20f-4f32-8bbd-f88a776b0b34\") " pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 11:59:58.810120 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.810040 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:58.811790 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.811775 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 11:59:58.811924 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.811907 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 11:59:58.820480 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:58.820458 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 15 11:59:58.820555 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:58.820513 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs podName:76443aa4-c20f-4f32-8bbd-f88a776b0b34 nodeName:}" failed. No retries permitted until 2026-07-15 12:01:02.820496099 +0000 UTC m=+130.234024428 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs") pod "network-metrics-daemon-l4d6j" (UID: "76443aa4-c20f-4f32-8bbd-f88a776b0b34") : secret "metrics-daemon-secret" not found Jul 15 11:59:58.822801 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.822774 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/977f5c84-a6ea-4927-93ad-9dd0e23853a8-original-pull-secret\") pod \"global-pull-secret-syncer-ggb88\" (UID: \"977f5c84-a6ea-4927-93ad-9dd0e23853a8\") " pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:58.911367 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.911268 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8rw8\" (UniqueName: \"kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8\") pod \"network-check-target-pd45b\" (UID: \"0c8aff4d-50ce-4f75-84fb-786fe725d46e\") " pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:58.913490 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.913471 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 11:59:58.923873 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.923850 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 11:59:58.934960 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:58.934930 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8rw8\" (UniqueName: \"kubernetes.io/projected/0c8aff4d-50ce-4f75-84fb-786fe725d46e-kube-api-access-f8rw8\") pod \"network-check-target-pd45b\" (UID: \"0c8aff4d-50ce-4f75-84fb-786fe725d46e\") " pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:59.001634 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:59.001597 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ggb88" Jul 15 11:59:59.011762 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:59.011732 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7c5z8\"" Jul 15 11:59:59.018064 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:59.018036 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 11:59:59.112617 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:59.112575 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:59.112794 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:59.112647 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 11:59:59.112794 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:59.112712 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 11:59:59.112794 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:59.112792 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 11:59:59.112939 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:59.112813 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm podName:144f5339-e458-46aa-acbe-f43a1e4dbec9 nodeName:}" failed. No retries permitted until 2026-07-15 12:00:01.112791312 +0000 UTC m=+68.526319647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-289vh" (UID: "144f5339-e458-46aa-acbe-f43a1e4dbec9") : configmap "kube-rbac-proxy" not found Jul 15 11:59:59.112939 ip-10-0-131-53 kubenswrapper[2570]: E0715 11:59:59.112850 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls podName:144f5339-e458-46aa-acbe-f43a1e4dbec9 nodeName:}" failed. No retries permitted until 2026-07-15 12:00:01.112831363 +0000 UTC m=+68.526359694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-289vh" (UID: "144f5339-e458-46aa-acbe-f43a1e4dbec9") : secret "insights-runtime-extractor-tls" not found Jul 15 11:59:59.133159 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:59.133130 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ggb88"] Jul 15 11:59:59.136121 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:59:59.136098 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977f5c84_a6ea_4927_93ad_9dd0e23853a8.slice/crio-872d7ea919ed40540274d5b99158495779a839889deb198ea418d86880f37132 WatchSource:0}: Error finding container 872d7ea919ed40540274d5b99158495779a839889deb198ea418d86880f37132: Status 404 returned error can't find the container with id 872d7ea919ed40540274d5b99158495779a839889deb198ea418d86880f37132 Jul 15 11:59:59.147612 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:59.147589 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pd45b"] Jul 15 11:59:59.150679 ip-10-0-131-53 kubenswrapper[2570]: W0715 11:59:59.150650 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8aff4d_50ce_4f75_84fb_786fe725d46e.slice/crio-977a1b98fb23c181554cdf14ee22f2e0cb45a4c8e8301765a2552f397e5ba515 WatchSource:0}: Error finding container 977a1b98fb23c181554cdf14ee22f2e0cb45a4c8e8301765a2552f397e5ba515: Status 404 returned error can't find the container with id 977a1b98fb23c181554cdf14ee22f2e0cb45a4c8e8301765a2552f397e5ba515 Jul 15 11:59:59.318053 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:59.317964 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pd45b" event={"ID":"0c8aff4d-50ce-4f75-84fb-786fe725d46e","Type":"ContainerStarted","Data":"977a1b98fb23c181554cdf14ee22f2e0cb45a4c8e8301765a2552f397e5ba515"} Jul 15 11:59:59.318921 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:59.318900 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ggb88" event={"ID":"977f5c84-a6ea-4927-93ad-9dd0e23853a8","Type":"ContainerStarted","Data":"872d7ea919ed40540274d5b99158495779a839889deb198ea418d86880f37132"} Jul 15 11:59:59.319884 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:59.319864 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq" event={"ID":"200a1940-87d2-4876-a7a3-06658dcd4ecb","Type":"ContainerStarted","Data":"faf567b9f042089afa0aabee3b92e9e9ec063c3862e15d2f6cebcddc2c95ce07"} Jul 15 11:59:59.406039 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:59.406004 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 11:59:59.494864 ip-10-0-131-53 kubenswrapper[2570]: I0715 11:59:59.494833 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:00.002833 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:00.002780 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9zcb8_3a23f0bc-7022-4e72-b3cf-0a63b6e681f8/dns-node-resolver/0.log" Jul 15 12:00:00.401620 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:00.401590 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:01.130293 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:01.129613 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 12:00:01.130293 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:01.129772 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 12:00:01.130293 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:01.129866 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 12:00:01.130293 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:01.129943 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm podName:144f5339-e458-46aa-acbe-f43a1e4dbec9 nodeName:}" failed. No retries permitted until 2026-07-15 12:00:05.129920684 +0000 UTC m=+72.543449033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-289vh" (UID: "144f5339-e458-46aa-acbe-f43a1e4dbec9") : configmap "kube-rbac-proxy" not found Jul 15 12:00:01.130293 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:01.129945 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 12:00:01.130293 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:01.130017 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls podName:144f5339-e458-46aa-acbe-f43a1e4dbec9 nodeName:}" failed. No retries permitted until 2026-07-15 12:00:05.129998207 +0000 UTC m=+72.543526538 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-289vh" (UID: "144f5339-e458-46aa-acbe-f43a1e4dbec9") : secret "insights-runtime-extractor-tls" not found Jul 15 12:00:01.202028 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:01.202003 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-snrv8_e507f9d6-cd28-4d22-9dfe-5869963a8c48/node-ca/0.log" Jul 15 12:00:01.331211 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:01.331181 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 12:00:01.331412 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:01.331360 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 12:00:01.331476 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:01.331438 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls podName:a92dead1-e314-46f5-9a12-c34efd573904 nodeName:}" failed. No retries permitted until 2026-07-15 12:00:33.331401489 +0000 UTC m=+100.744929819 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls") pod "dns-default-24s4z" (UID: "a92dead1-e314-46f5-9a12-c34efd573904") : secret "dns-default-metrics-tls" not found Jul 15 12:00:01.801436 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:01.801399 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5zvqc_8b959ddc-0f8a-473d-a311-afcfa283fed3/serve-healthcheck-canary/0.log" Jul 15 12:00:01.986376 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:01.986229 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:02.574155 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:02.574123 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:02.604338 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:02.604307 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-ftmvb_74bcecd1-b162-4ce3-b731-d47b223860a8/kube-storage-version-migrator-operator/0.log" Jul 15 12:00:02.688426 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:02.688356 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:02.789161 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:02.789118 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:03.339920 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:03.339877 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:03.423021 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:03.422977 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:03.944315 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:03.944274 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:04.100758 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:04.100712 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:04.168761 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:04.168712 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:05.158123 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:05.158092 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:05.162122 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:05.162102 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 12:00:05.162229 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:05.162175 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 12:00:05.162267 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:05.162234 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 12:00:05.162267 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:05.162253 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 12:00:05.162424 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:05.162294 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm podName:144f5339-e458-46aa-acbe-f43a1e4dbec9 nodeName:}" failed. No retries permitted until 2026-07-15 12:00:13.162280982 +0000 UTC m=+80.575809311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-289vh" (UID: "144f5339-e458-46aa-acbe-f43a1e4dbec9") : configmap "kube-rbac-proxy" not found Jul 15 12:00:05.162424 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:05.162307 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls podName:144f5339-e458-46aa-acbe-f43a1e4dbec9 nodeName:}" failed. No retries permitted until 2026-07-15 12:00:13.162301142 +0000 UTC m=+80.575829471 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-289vh" (UID: "144f5339-e458-46aa-acbe-f43a1e4dbec9") : secret "insights-runtime-extractor-tls" not found Jul 15 12:00:05.341952 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:05.341906 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ggb88" event={"ID":"977f5c84-a6ea-4927-93ad-9dd0e23853a8","Type":"ContainerStarted","Data":"382d867f5bad921ce54b07fb0ebcb44996b9275acfed5f20340219aed7ce545d"} Jul 15 12:00:05.343316 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:05.343290 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq" event={"ID":"200a1940-87d2-4876-a7a3-06658dcd4ecb","Type":"ContainerStarted","Data":"d013dc51298c8e96b0f78209295539bf8faa8d9547f6d179fd6e84fdaccf4f12"} Jul 15 12:00:05.344636 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:05.344612 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pd45b" event={"ID":"0c8aff4d-50ce-4f75-84fb-786fe725d46e","Type":"ContainerStarted","Data":"2679a2076ecdd8a261ea172d8cf4d90ec887b053988d9cb68946a5f0b8d79cfc"} Jul 15 12:00:05.344776 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:05.344760 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 12:00:05.381960 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:05.381911 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ggb88" podStartSLOduration=66.702797313 podStartE2EDuration="1m12.381894762s" totalImagesPullingTime="5.679097449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:00:05.363368912 +0000 UTC m=+72.776897264" watchObservedRunningTime="2026-07-15 12:00:05.381894762 +0000 UTC m=+72.795423116" Jul 15 12:00:05.382548 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:05.382520 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pd45b" podStartSLOduration=66.723541667 podStartE2EDuration="1m12.382511204s" totalImagesPullingTime="5.658969537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:00:05.380774343 +0000 UTC m=+72.794302707" watchObservedRunningTime="2026-07-15 12:00:05.382511204 +0000 UTC m=+72.796039586" Jul 15 12:00:05.406481 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:05.406439 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cd8d4fbb6-tfgwq" podStartSLOduration=1.3921086169999999 podStartE2EDuration="7.406424026s" totalImagesPullingTime="6.014315409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:00:05.406335716 +0000 UTC m=+72.819864085" watchObservedRunningTime="2026-07-15 12:00:05.406424026 +0000 UTC m=+72.819952377" Jul 15 12:00:05.759695 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:05.759664 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:06.420345 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:06.420304 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:06.521544 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:06.521510 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:06.693317 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:06.693210 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:06.775476 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:06.775440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 12:00:06.775626 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:06.775532 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 12:00:06.775713 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:06.775621 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle podName:f8699bdf-f732-4921-a592-483bea1c84fb nodeName:}" failed. No retries permitted until 2026-07-15 12:00:22.775598544 +0000 UTC m=+90.189126874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle") pod "router-default-5c864b74c8-z6ccx" (UID: "f8699bdf-f732-4921-a592-483bea1c84fb") : configmap references non-existent config key: service-ca.crt Jul 15 12:00:06.778057 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:06.778037 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8699bdf-f732-4921-a592-483bea1c84fb-metrics-certs\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 12:00:06.896649 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:06.896620 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:07.532549 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:07.532511 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:07.703538 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:07.703502 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:07.780544 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:07.780509 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:08.140842 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:08.140805 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:08.275703 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:08.275675 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:08.692663 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:08.692628 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:09.282907 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:09.282870 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:09.486869 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:09.486835 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:09.684878 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:09.684840 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:09.891992 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:09.891961 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:11.450696 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:11.450657 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:11.551135 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:11.551095 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:11.694312 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:11.694275 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:12.089052 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:12.089024 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:12.493344 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:12.493301 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:12.692774 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:12.692721 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:13.226886 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:13.226854 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 12:00:13.227040 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:13.226931 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 12:00:13.227369 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:13.227351 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/144f5339-e458-46aa-acbe-f43a1e4dbec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 12:00:13.229521 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:13.229495 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/144f5339-e458-46aa-acbe-f43a1e4dbec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-289vh\" (UID: \"144f5339-e458-46aa-acbe-f43a1e4dbec9\") " pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 12:00:13.321959 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:13.321919 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-289vh" Jul 15 12:00:13.445512 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:13.445485 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-289vh"] Jul 15 12:00:13.448783 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:00:13.448758 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144f5339_e458_46aa_acbe_f43a1e4dbec9.slice/crio-796d7b37755d91e2a559b84564260c679a875e0299b9390ab79ab004f7907146 WatchSource:0}: Error finding container 796d7b37755d91e2a559b84564260c679a875e0299b9390ab79ab004f7907146: Status 404 returned error can't find the container with id 796d7b37755d91e2a559b84564260c679a875e0299b9390ab79ab004f7907146 Jul 15 12:00:14.378728 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:14.378613 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-289vh" event={"ID":"144f5339-e458-46aa-acbe-f43a1e4dbec9","Type":"ContainerStarted","Data":"796d7b37755d91e2a559b84564260c679a875e0299b9390ab79ab004f7907146"} Jul 15 12:00:16.704412 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:16.704351 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:16.770679 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:16.770633 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:17.440951 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:17.440902 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:17.542676 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:17.542627 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:17.649766 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:17.649710 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:17.726040 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:17.725950 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:17.873364 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:17.873327 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:18.043877 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:18.043779 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:18.882536 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:18.882483 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:19.400654 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:19.400621 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-289vh" event={"ID":"144f5339-e458-46aa-acbe-f43a1e4dbec9","Type":"ContainerStarted","Data":"a8433766c530971fa4d0cd34dc5bdea328a4164b56383baa0e4e214c9f54a6f0"} Jul 15 12:00:20.974866 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:20.974829 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:21.036298 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.036263 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:21.589757 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.589717 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d6cc95958-zp5fl"] Jul 15 12:00:21.603848 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.603814 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.606854 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.606820 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 15 12:00:21.607098 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.607055 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 15 12:00:21.607340 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.607319 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 15 12:00:21.607618 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.607597 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vl9cw\"" Jul 15 12:00:21.607959 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.607931 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d6cc95958-zp5fl"] Jul 15 12:00:21.612085 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.612061 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 15 12:00:21.636375 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.636344 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d6cc95958-zp5fl"] Jul 15 12:00:21.636598 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:21.636574 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-g4qr8 registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-g4qr8 registry-certificates registry-tls trusted-ca]: context canceled" pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" podUID="bb4baff8-a029-4113-ac96-9eaab77fe169" Jul 15 12:00:21.696557 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.696515 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-registry-tls\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.696763 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.696579 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4qr8\" (UniqueName: \"kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-kube-api-access-g4qr8\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.696763 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.696617 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb4baff8-a029-4113-ac96-9eaab77fe169-image-registry-private-configuration\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.696763 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.696653 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb4baff8-a029-4113-ac96-9eaab77fe169-registry-certificates\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.696979 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.696810 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb4baff8-a029-4113-ac96-9eaab77fe169-ca-trust-extracted\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.696979 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.696844 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb4baff8-a029-4113-ac96-9eaab77fe169-installation-pull-secrets\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.696979 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.696872 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-bound-sa-token\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.696979 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.696959 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb4baff8-a029-4113-ac96-9eaab77fe169-trusted-ca\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.709677 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.709578 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:21.798193 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.798163 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-registry-tls\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.798389 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.798217 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4qr8\" (UniqueName: \"kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-kube-api-access-g4qr8\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.798389 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.798252 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb4baff8-a029-4113-ac96-9eaab77fe169-image-registry-private-configuration\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.798389 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.798273 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb4baff8-a029-4113-ac96-9eaab77fe169-registry-certificates\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.798570 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.798465 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb4baff8-a029-4113-ac96-9eaab77fe169-ca-trust-extracted\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.798570 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.798514 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb4baff8-a029-4113-ac96-9eaab77fe169-installation-pull-secrets\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.798570 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.798545 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-bound-sa-token\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.798765 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.798592 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb4baff8-a029-4113-ac96-9eaab77fe169-trusted-ca\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.798857 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.798833 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb4baff8-a029-4113-ac96-9eaab77fe169-ca-trust-extracted\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.799413 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.799380 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb4baff8-a029-4113-ac96-9eaab77fe169-registry-certificates\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.799413 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.799380 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:21.799996 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.799975 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb4baff8-a029-4113-ac96-9eaab77fe169-trusted-ca\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.801170 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.801142 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb4baff8-a029-4113-ac96-9eaab77fe169-installation-pull-secrets\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.801509 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.801489 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb4baff8-a029-4113-ac96-9eaab77fe169-image-registry-private-configuration\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.801560 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.801493 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-registry-tls\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.809379 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.809345 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4qr8\" (UniqueName: \"kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-kube-api-access-g4qr8\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:21.810824 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:21.810799 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-bound-sa-token\") pod \"image-registry-5d6cc95958-zp5fl\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:22.412345 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.412294 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-289vh" event={"ID":"144f5339-e458-46aa-acbe-f43a1e4dbec9","Type":"ContainerStarted","Data":"e67d33ef8db1db75379f1819d687d993c5c91ceb89076a9e693c655e2449d03c"} Jul 15 12:00:22.412815 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.412315 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:22.418150 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.418131 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:22.504693 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.504653 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-bound-sa-token\") pod \"bb4baff8-a029-4113-ac96-9eaab77fe169\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " Jul 15 12:00:22.504888 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.504716 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb4baff8-a029-4113-ac96-9eaab77fe169-image-registry-private-configuration\") pod \"bb4baff8-a029-4113-ac96-9eaab77fe169\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " Jul 15 12:00:22.504888 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.504766 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb4baff8-a029-4113-ac96-9eaab77fe169-registry-certificates\") pod \"bb4baff8-a029-4113-ac96-9eaab77fe169\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " Jul 15 12:00:22.504888 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.504799 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb4baff8-a029-4113-ac96-9eaab77fe169-ca-trust-extracted\") pod \"bb4baff8-a029-4113-ac96-9eaab77fe169\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " Jul 15 12:00:22.504888 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.504832 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4qr8\" (UniqueName: \"kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-kube-api-access-g4qr8\") pod \"bb4baff8-a029-4113-ac96-9eaab77fe169\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " Jul 15 12:00:22.504888 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.504856 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb4baff8-a029-4113-ac96-9eaab77fe169-installation-pull-secrets\") pod \"bb4baff8-a029-4113-ac96-9eaab77fe169\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " Jul 15 12:00:22.505127 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.504893 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb4baff8-a029-4113-ac96-9eaab77fe169-trusted-ca\") pod \"bb4baff8-a029-4113-ac96-9eaab77fe169\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " Jul 15 12:00:22.505127 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.504918 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-registry-tls\") pod \"bb4baff8-a029-4113-ac96-9eaab77fe169\" (UID: \"bb4baff8-a029-4113-ac96-9eaab77fe169\") " Jul 15 12:00:22.505226 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.505142 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb4baff8-a029-4113-ac96-9eaab77fe169-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bb4baff8-a029-4113-ac96-9eaab77fe169" (UID: "bb4baff8-a029-4113-ac96-9eaab77fe169"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:00:22.505401 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.505345 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4baff8-a029-4113-ac96-9eaab77fe169-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bb4baff8-a029-4113-ac96-9eaab77fe169" (UID: "bb4baff8-a029-4113-ac96-9eaab77fe169"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:00:22.505521 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.505395 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb4baff8-a029-4113-ac96-9eaab77fe169-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bb4baff8-a029-4113-ac96-9eaab77fe169" (UID: "bb4baff8-a029-4113-ac96-9eaab77fe169"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:00:22.507337 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.507310 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb4baff8-a029-4113-ac96-9eaab77fe169-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bb4baff8-a029-4113-ac96-9eaab77fe169" (UID: "bb4baff8-a029-4113-ac96-9eaab77fe169"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:00:22.507337 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.507322 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-kube-api-access-g4qr8" (OuterVolumeSpecName: "kube-api-access-g4qr8") pod "bb4baff8-a029-4113-ac96-9eaab77fe169" (UID: "bb4baff8-a029-4113-ac96-9eaab77fe169"). InnerVolumeSpecName "kube-api-access-g4qr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:00:22.507466 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.507387 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bb4baff8-a029-4113-ac96-9eaab77fe169" (UID: "bb4baff8-a029-4113-ac96-9eaab77fe169"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:00:22.507520 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.507495 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb4baff8-a029-4113-ac96-9eaab77fe169-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bb4baff8-a029-4113-ac96-9eaab77fe169" (UID: "bb4baff8-a029-4113-ac96-9eaab77fe169"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:00:22.507520 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.507499 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bb4baff8-a029-4113-ac96-9eaab77fe169" (UID: "bb4baff8-a029-4113-ac96-9eaab77fe169"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:00:22.606175 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.606133 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-bound-sa-token\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:00:22.606175 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.606165 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bb4baff8-a029-4113-ac96-9eaab77fe169-image-registry-private-configuration\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:00:22.606175 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.606177 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bb4baff8-a029-4113-ac96-9eaab77fe169-registry-certificates\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:00:22.606175 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.606187 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bb4baff8-a029-4113-ac96-9eaab77fe169-ca-trust-extracted\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:00:22.606434 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.606197 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4qr8\" (UniqueName: \"kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-kube-api-access-g4qr8\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:00:22.606434 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.606205 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bb4baff8-a029-4113-ac96-9eaab77fe169-installation-pull-secrets\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:00:22.606434 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.606213 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bb4baff8-a029-4113-ac96-9eaab77fe169-trusted-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:00:22.606434 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.606222 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bb4baff8-a029-4113-ac96-9eaab77fe169-registry-tls\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:00:22.808027 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.808000 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 12:00:22.808595 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.808576 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8699bdf-f732-4921-a592-483bea1c84fb-service-ca-bundle\") pod \"router-default-5c864b74c8-z6ccx\" (UID: \"f8699bdf-f732-4921-a592-483bea1c84fb\") " pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 12:00:22.970867 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.970836 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2q8rk\"" Jul 15 12:00:22.979280 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:22.979261 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 12:00:23.112472 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.112441 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c864b74c8-z6ccx"] Jul 15 12:00:23.116233 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:00:23.116197 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8699bdf_f732_4921_a592_483bea1c84fb.slice/crio-9f2bc5d06a198db0a371bbcecbf2cf7d246648ab5d400c23a78c54b038f4c877 WatchSource:0}: Error finding container 9f2bc5d06a198db0a371bbcecbf2cf7d246648ab5d400c23a78c54b038f4c877: Status 404 returned error can't find the container with id 9f2bc5d06a198db0a371bbcecbf2cf7d246648ab5d400c23a78c54b038f4c877 Jul 15 12:00:23.422910 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.422882 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d6cc95958-zp5fl" Jul 15 12:00:23.422910 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.422896 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c864b74c8-z6ccx" event={"ID":"f8699bdf-f732-4921-a592-483bea1c84fb","Type":"ContainerStarted","Data":"bb3eaad2d8a6c4a704b192b83e788d2d1c992f82c9be2c25a2a1c130f9f60b60"} Jul 15 12:00:23.423396 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.422971 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c864b74c8-z6ccx" event={"ID":"f8699bdf-f732-4921-a592-483bea1c84fb","Type":"ContainerStarted","Data":"9f2bc5d06a198db0a371bbcecbf2cf7d246648ab5d400c23a78c54b038f4c877"} Jul 15 12:00:23.448211 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.448151 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5c864b74c8-z6ccx" podStartSLOduration=33.448131422 podStartE2EDuration="33.448131422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:00:23.446500918 +0000 UTC m=+90.860029402" watchObservedRunningTime="2026-07-15 12:00:23.448131422 +0000 UTC m=+90.861659777" Jul 15 12:00:23.485896 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.485861 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d6cc95958-zp5fl"] Jul 15 12:00:23.488616 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.488592 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78db486f6-mpwwc"] Jul 15 12:00:23.491818 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.491797 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.493917 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.493891 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 12:00:23.494035 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.493953 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 12:00:23.494095 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.493953 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 12:00:23.494628 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.494334 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 12:00:23.494628 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.494342 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 12:00:23.494628 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.494372 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 12:00:23.494628 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.494375 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 12:00:23.494628 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.494377 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p4glj\"" Jul 15 12:00:23.500255 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.500232 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d6cc95958-zp5fl"] Jul 15 12:00:23.502809 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.502788 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78db486f6-mpwwc"] Jul 15 12:00:23.613905 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.613868 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-service-ca\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.614185 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.613922 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-config\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.614185 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.614037 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv5q9\" (UniqueName: \"kubernetes.io/projected/a8212cbc-ce8c-4c5b-9976-e76c68a70808-kube-api-access-kv5q9\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.614185 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.614073 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-serving-cert\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.614185 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.614130 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-oauth-config\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.614185 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.614160 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-oauth-serving-cert\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.714908 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.714819 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-service-ca\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.714908 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.714858 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-config\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.715185 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.714995 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv5q9\" (UniqueName: \"kubernetes.io/projected/a8212cbc-ce8c-4c5b-9976-e76c68a70808-kube-api-access-kv5q9\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.715185 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.715042 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-serving-cert\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.715185 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.715114 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-oauth-config\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.715185 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.715149 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-oauth-serving-cert\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.715690 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.715642 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-service-ca\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.715690 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.715642 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-config\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.715878 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.715823 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-oauth-serving-cert\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.718097 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.718072 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-serving-cert\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.718234 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.718209 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-oauth-config\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.722898 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.722869 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv5q9\" (UniqueName: \"kubernetes.io/projected/a8212cbc-ce8c-4c5b-9976-e76c68a70808-kube-api-access-kv5q9\") pod \"console-78db486f6-mpwwc\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.803090 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.802852 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:23.876793 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.876672 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:23.947969 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.947930 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78db486f6-mpwwc"] Jul 15 12:00:23.952267 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:00:23.952235 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8212cbc_ce8c_4c5b_9976_e76c68a70808.slice/crio-118267eaf80de58aebc33f810fa0f248c33d6c809f9a139621dd5db13a52a55a WatchSource:0}: Error finding container 118267eaf80de58aebc33f810fa0f248c33d6c809f9a139621dd5db13a52a55a: Status 404 returned error can't find the container with id 118267eaf80de58aebc33f810fa0f248c33d6c809f9a139621dd5db13a52a55a Jul 15 12:00:23.979798 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.979711 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 12:00:23.982613 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:23.982585 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 12:00:24.427286 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:24.427250 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78db486f6-mpwwc" event={"ID":"a8212cbc-ce8c-4c5b-9976-e76c68a70808","Type":"ContainerStarted","Data":"118267eaf80de58aebc33f810fa0f248c33d6c809f9a139621dd5db13a52a55a"} Jul 15 12:00:24.427718 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:24.427485 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 12:00:24.428929 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:24.428904 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5c864b74c8-z6ccx" Jul 15 12:00:24.716396 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:24.716300 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:24.780129 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:24.780095 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:25.090663 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:25.090589 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb4baff8-a029-4113-ac96-9eaab77fe169" path="/var/lib/kubelet/pods/bb4baff8-a029-4113-ac96-9eaab77fe169/volumes" Jul 15 12:00:26.436939 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:26.436889 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-289vh" event={"ID":"144f5339-e458-46aa-acbe-f43a1e4dbec9","Type":"ContainerStarted","Data":"c17da6914f8957f4a1e16aa7e88fea6061ae3844006bf081657b01206310cf9c"} Jul 15 12:00:26.456647 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:26.456030 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-289vh" podStartSLOduration=16.789039457 podStartE2EDuration="29.456011917s" totalImagesPullingTime="12.66697246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 11:59:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:00:26.455760044 +0000 UTC m=+93.869288388" watchObservedRunningTime="2026-07-15 12:00:26.456011917 +0000 UTC m=+93.869540269" Jul 15 12:00:27.033691 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:27.033650 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:27.139523 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:27.139476 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:27.232478 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:27.232444 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:27.664139 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:27.664104 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:28.486653 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:28.486617 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:28.566724 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:28.566684 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:29.448874 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:29.448840 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78db486f6-mpwwc" event={"ID":"a8212cbc-ce8c-4c5b-9976-e76c68a70808","Type":"ContainerStarted","Data":"640e0f05cf860ebd95f496286774cff83da2f9bdd5e53a69c14b91a4225a409d"} Jul 15 12:00:29.466725 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:29.466674 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78db486f6-mpwwc" podStartSLOduration=1.5065978979999999 podStartE2EDuration="6.466658339s" totalImagesPullingTime="4.960060441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:00:29.466120828 +0000 UTC m=+96.879649182" watchObservedRunningTime="2026-07-15 12:00:29.466658339 +0000 UTC m=+96.880186689" Jul 15 12:00:31.252560 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:31.252525 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:31.347612 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:31.347575 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:31.789449 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:31.789416 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:33.397654 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:33.397608 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 12:00:33.400036 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:33.399994 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a92dead1-e314-46f5-9a12-c34efd573904-metrics-tls\") pod \"dns-default-24s4z\" (UID: \"a92dead1-e314-46f5-9a12-c34efd573904\") " pod="openshift-dns/dns-default-24s4z" Jul 15 12:00:33.644495 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:33.644464 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bqsqx\"" Jul 15 12:00:33.653096 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:33.653043 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-24s4z" Jul 15 12:00:33.779352 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:33.779320 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-24s4z"] Jul 15 12:00:33.781967 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:00:33.781940 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92dead1_e314_46f5_9a12_c34efd573904.slice/crio-fc83aa5dd0ad7817cb0baa8930753c4cdb1fd7a815b458e2fd4f3e6bfad5d585 WatchSource:0}: Error finding container fc83aa5dd0ad7817cb0baa8930753c4cdb1fd7a815b458e2fd4f3e6bfad5d585: Status 404 returned error can't find the container with id fc83aa5dd0ad7817cb0baa8930753c4cdb1fd7a815b458e2fd4f3e6bfad5d585 Jul 15 12:00:33.803073 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:33.803045 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:33.803161 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:33.803103 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:00:33.804368 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:33.804339 2570 patch_prober.go:28] interesting pod/console-78db486f6-mpwwc container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.13:8443/health\": dial tcp 10.133.0.13:8443: connect: connection refused" start-of-body= Jul 15 12:00:33.804417 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:33.804382 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-78db486f6-mpwwc" podUID="a8212cbc-ce8c-4c5b-9976-e76c68a70808" containerName="console" probeResult="failure" output="Get \"https://10.133.0.13:8443/health\": dial tcp 10.133.0.13:8443: connect: connection refused" Jul 15 12:00:34.466978 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:34.466945 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-24s4z" event={"ID":"a92dead1-e314-46f5-9a12-c34efd573904","Type":"ContainerStarted","Data":"fc83aa5dd0ad7817cb0baa8930753c4cdb1fd7a815b458e2fd4f3e6bfad5d585"} Jul 15 12:00:34.632580 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:34.632544 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:34.732650 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:34.732569 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:35.487981 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:35.487935 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:35.588146 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:35.588113 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:35.788666 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:35.788624 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:36.350120 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:36.350093 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pd45b" Jul 15 12:00:36.475774 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:36.475722 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-24s4z" event={"ID":"a92dead1-e314-46f5-9a12-c34efd573904","Type":"ContainerStarted","Data":"26d438692dcef1e423c69f8e7a7ca72f8f778d46e3e6d7fbee2dba697d542e77"} Jul 15 12:00:36.475938 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:36.475908 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-24s4z" Jul 15 12:00:36.475984 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:36.475945 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-24s4z" event={"ID":"a92dead1-e314-46f5-9a12-c34efd573904","Type":"ContainerStarted","Data":"a49d5db3110d4dcea1f554ad97cdf4a855997c4cd2d08bf218ce3e5867c06e77"} Jul 15 12:00:36.492279 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:36.492231 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-24s4z" podStartSLOduration=65.682370912 podStartE2EDuration="1m7.492219062s" totalImagesPullingTime="1.80984815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:00:36.491944546 +0000 UTC m=+103.905472898" watchObservedRunningTime="2026-07-15 12:00:36.492219062 +0000 UTC m=+103.905747406" Jul 15 12:00:37.495491 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.495457 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:37.607685 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.607654 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:37.631300 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.631270 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj"] Jul 15 12:00:37.678548 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.678516 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj"] Jul 15 12:00:37.678704 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.678562 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-nwkhg"] Jul 15 12:00:37.678704 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.678642 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" Jul 15 12:00:37.680785 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.680756 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 12:00:37.680785 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.680759 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 12:00:37.680965 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.680763 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 12:00:37.680965 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.680921 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rxb2v\"" Jul 15 12:00:37.681055 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.681024 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 15 12:00:37.681121 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.681100 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 15 12:00:37.691941 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.691922 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-chq6t"] Jul 15 12:00:37.692050 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.692041 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.693910 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.693892 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 15 12:00:37.694019 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.693950 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 15 12:00:37.694105 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.694085 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 15 12:00:37.694156 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.694085 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8t6sb\"" Jul 15 12:00:37.704331 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.704312 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:37.713257 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.713240 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-nwkhg"] Jul 15 12:00:37.713364 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.713351 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.715672 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.715634 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 12:00:37.715788 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.715730 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 12:00:37.715836 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.715825 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 12:00:37.715876 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.715851 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-262w7\"" Jul 15 12:00:37.834575 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.834481 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/279e04c2-80d5-4f93-83b0-fb8ab35223f0-metrics-client-ca\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.834575 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.834517 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-tls\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.834575 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.834537 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9924fcc-57eb-421b-8255-4ee0fbb198cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.834575 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.834561 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.834882 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.834631 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f5dk\" (UniqueName: \"kubernetes.io/projected/279e04c2-80d5-4f93-83b0-fb8ab35223f0-kube-api-access-4f5dk\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.834882 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.834688 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-textfile\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.834882 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.834712 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f9924fcc-57eb-421b-8255-4ee0fbb198cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.834882 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.834781 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54e8bb88-05f0-4569-968e-93f700263ce5-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-qwrhj\" (UID: \"54e8bb88-05f0-4569-968e-93f700263ce5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" Jul 15 12:00:37.834882 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.834807 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-accelerators-collector-config\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.834882 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.834825 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9924fcc-57eb-421b-8255-4ee0fbb198cb-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.834882 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.834843 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f9924fcc-57eb-421b-8255-4ee0fbb198cb-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.834882 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.834871 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54e8bb88-05f0-4569-968e-93f700263ce5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-qwrhj\" (UID: \"54e8bb88-05f0-4569-968e-93f700263ce5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" Jul 15 12:00:37.835143 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.834897 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2rqk\" (UniqueName: \"kubernetes.io/projected/54e8bb88-05f0-4569-968e-93f700263ce5-kube-api-access-c2rqk\") pod \"openshift-state-metrics-84bd68b896-qwrhj\" (UID: \"54e8bb88-05f0-4569-968e-93f700263ce5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" Jul 15 12:00:37.835143 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.834915 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-wtmp\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.835143 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.834948 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ph7\" (UniqueName: \"kubernetes.io/projected/f9924fcc-57eb-421b-8255-4ee0fbb198cb-kube-api-access-x7ph7\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.835143 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.835016 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9924fcc-57eb-421b-8255-4ee0fbb198cb-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.835143 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.835049 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/279e04c2-80d5-4f93-83b0-fb8ab35223f0-sys\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.835143 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.835070 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/279e04c2-80d5-4f93-83b0-fb8ab35223f0-root\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.835143 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.835094 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8bb88-05f0-4569-968e-93f700263ce5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-qwrhj\" (UID: \"54e8bb88-05f0-4569-968e-93f700263ce5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" Jul 15 12:00:37.846639 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.846609 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:37.935603 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935567 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/279e04c2-80d5-4f93-83b0-fb8ab35223f0-sys\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.935603 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935604 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/279e04c2-80d5-4f93-83b0-fb8ab35223f0-root\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.935891 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935629 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8bb88-05f0-4569-968e-93f700263ce5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-qwrhj\" (UID: \"54e8bb88-05f0-4569-968e-93f700263ce5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" Jul 15 12:00:37.935891 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935654 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/279e04c2-80d5-4f93-83b0-fb8ab35223f0-metrics-client-ca\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.935891 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935682 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-tls\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.935891 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935690 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/279e04c2-80d5-4f93-83b0-fb8ab35223f0-sys\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.935891 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935705 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9924fcc-57eb-421b-8255-4ee0fbb198cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.935891 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935704 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/279e04c2-80d5-4f93-83b0-fb8ab35223f0-root\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.935891 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935758 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.935891 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935787 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f5dk\" (UniqueName: \"kubernetes.io/projected/279e04c2-80d5-4f93-83b0-fb8ab35223f0-kube-api-access-4f5dk\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.935891 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935803 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-textfile\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.935891 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:37.935808 2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 15 12:00:37.935891 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935818 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f9924fcc-57eb-421b-8255-4ee0fbb198cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.935891 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:37.935868 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-tls podName:279e04c2-80d5-4f93-83b0-fb8ab35223f0 nodeName:}" failed. No retries permitted until 2026-07-15 12:00:38.43584781 +0000 UTC m=+105.849376154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-tls") pod "node-exporter-chq6t" (UID: "279e04c2-80d5-4f93-83b0-fb8ab35223f0") : secret "node-exporter-tls" not found Jul 15 12:00:37.935891 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935887 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54e8bb88-05f0-4569-968e-93f700263ce5-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-qwrhj\" (UID: \"54e8bb88-05f0-4569-968e-93f700263ce5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" Jul 15 12:00:37.936513 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935923 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-accelerators-collector-config\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.936513 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935952 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9924fcc-57eb-421b-8255-4ee0fbb198cb-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.936513 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.935982 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f9924fcc-57eb-421b-8255-4ee0fbb198cb-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.936513 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.936021 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54e8bb88-05f0-4569-968e-93f700263ce5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-qwrhj\" (UID: \"54e8bb88-05f0-4569-968e-93f700263ce5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" Jul 15 12:00:37.936513 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.936048 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2rqk\" (UniqueName: \"kubernetes.io/projected/54e8bb88-05f0-4569-968e-93f700263ce5-kube-api-access-c2rqk\") pod \"openshift-state-metrics-84bd68b896-qwrhj\" (UID: \"54e8bb88-05f0-4569-968e-93f700263ce5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" Jul 15 12:00:37.936513 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.936075 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-wtmp\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.936513 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.936119 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ph7\" (UniqueName: \"kubernetes.io/projected/f9924fcc-57eb-421b-8255-4ee0fbb198cb-kube-api-access-x7ph7\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.936513 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.936163 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9924fcc-57eb-421b-8255-4ee0fbb198cb-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.936513 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.936170 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-textfile\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.936513 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:37.936300 2570 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 15 12:00:37.936513 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:37.936349 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9924fcc-57eb-421b-8255-4ee0fbb198cb-kube-state-metrics-tls podName:f9924fcc-57eb-421b-8255-4ee0fbb198cb nodeName:}" failed. No retries permitted until 2026-07-15 12:00:38.43633314 +0000 UTC m=+105.849861474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f9924fcc-57eb-421b-8255-4ee0fbb198cb-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-nwkhg" (UID: "f9924fcc-57eb-421b-8255-4ee0fbb198cb") : secret "kube-state-metrics-tls" not found Jul 15 12:00:37.936513 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.936393 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/279e04c2-80d5-4f93-83b0-fb8ab35223f0-metrics-client-ca\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.937148 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.936524 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-wtmp\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.937148 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.936798 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f9924fcc-57eb-421b-8255-4ee0fbb198cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.937252 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.937169 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54e8bb88-05f0-4569-968e-93f700263ce5-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-qwrhj\" (UID: \"54e8bb88-05f0-4569-968e-93f700263ce5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" Jul 15 12:00:37.937322 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.937257 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-accelerators-collector-config\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.937396 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.937345 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f9924fcc-57eb-421b-8255-4ee0fbb198cb-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.937559 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.937535 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9924fcc-57eb-421b-8255-4ee0fbb198cb-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.938441 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.938418 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.938756 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.938719 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9924fcc-57eb-421b-8255-4ee0fbb198cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.939040 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.939025 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/54e8bb88-05f0-4569-968e-93f700263ce5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-qwrhj\" (UID: \"54e8bb88-05f0-4569-968e-93f700263ce5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" Jul 15 12:00:37.939322 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.939305 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54e8bb88-05f0-4569-968e-93f700263ce5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-qwrhj\" (UID: \"54e8bb88-05f0-4569-968e-93f700263ce5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" Jul 15 12:00:37.944016 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.943983 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f5dk\" (UniqueName: \"kubernetes.io/projected/279e04c2-80d5-4f93-83b0-fb8ab35223f0-kube-api-access-4f5dk\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:37.947365 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.947339 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:37.948249 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.948228 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ph7\" (UniqueName: \"kubernetes.io/projected/f9924fcc-57eb-421b-8255-4ee0fbb198cb-kube-api-access-x7ph7\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:37.948594 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.948288 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2rqk\" (UniqueName: \"kubernetes.io/projected/54e8bb88-05f0-4569-968e-93f700263ce5-kube-api-access-c2rqk\") pod \"openshift-state-metrics-84bd68b896-qwrhj\" (UID: \"54e8bb88-05f0-4569-968e-93f700263ce5\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" Jul 15 12:00:37.988042 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:37.988005 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" Jul 15 12:00:38.046909 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.046880 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:38.148919 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.148886 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj"] Jul 15 12:00:38.151475 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:00:38.151448 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e8bb88_05f0_4569_968e_93f700263ce5.slice/crio-5592a39f5380c9675b9676f0489fe278de1f01c1fdacfa086a524be337271a77 WatchSource:0}: Error finding container 5592a39f5380c9675b9676f0489fe278de1f01c1fdacfa086a524be337271a77: Status 404 returned error can't find the container with id 5592a39f5380c9675b9676f0489fe278de1f01c1fdacfa086a524be337271a77 Jul 15 12:00:38.281731 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.281702 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:38.436471 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.436417 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:38.440520 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.440494 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-tls\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:38.440646 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.440550 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9924fcc-57eb-421b-8255-4ee0fbb198cb-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:38.442982 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.442959 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/279e04c2-80d5-4f93-83b0-fb8ab35223f0-node-exporter-tls\") pod \"node-exporter-chq6t\" (UID: \"279e04c2-80d5-4f93-83b0-fb8ab35223f0\") " pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:38.443074 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.443004 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9924fcc-57eb-421b-8255-4ee0fbb198cb-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-nwkhg\" (UID: \"f9924fcc-57eb-421b-8255-4ee0fbb198cb\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:38.483952 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.483912 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" event={"ID":"54e8bb88-05f0-4569-968e-93f700263ce5","Type":"ContainerStarted","Data":"5488d047bb3c9c8143f73775b5ca28e03f4a6bd0e296b776e0fc00f422bfb3ed"} Jul 15 12:00:38.484084 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.483969 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" event={"ID":"54e8bb88-05f0-4569-968e-93f700263ce5","Type":"ContainerStarted","Data":"33ccdaa849415b0568d0a166234db0257ffd3dc24b4f21c7a58c03dedbcde127"} Jul 15 12:00:38.484084 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.483988 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" event={"ID":"54e8bb88-05f0-4569-968e-93f700263ce5","Type":"ContainerStarted","Data":"5592a39f5380c9675b9676f0489fe278de1f01c1fdacfa086a524be337271a77"} Jul 15 12:00:38.542073 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.542048 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:38.600347 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.600320 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" Jul 15 12:00:38.622030 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.622007 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-chq6t" Jul 15 12:00:38.706531 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.706500 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:38.760163 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.760134 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-nwkhg"] Jul 15 12:00:38.763207 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:00:38.763184 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9924fcc_57eb_421b_8255_4ee0fbb198cb.slice/crio-6b7093f5fe581ae590a7434119ee7c00cad67515c8b24be0941e525647a0c993 WatchSource:0}: Error finding container 6b7093f5fe581ae590a7434119ee7c00cad67515c8b24be0941e525647a0c993: Status 404 returned error can't find the container with id 6b7093f5fe581ae590a7434119ee7c00cad67515c8b24be0941e525647a0c993 Jul 15 12:00:38.905728 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:38.905637 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:39.489948 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:39.489908 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chq6t" event={"ID":"279e04c2-80d5-4f93-83b0-fb8ab35223f0","Type":"ContainerStarted","Data":"bb9e2870ce58dd7681ba97118d50906934d546d2caae4037b5c65fe45e2c6350"} Jul 15 12:00:39.491208 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:39.491175 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" event={"ID":"f9924fcc-57eb-421b-8255-4ee0fbb198cb","Type":"ContainerStarted","Data":"6b7093f5fe581ae590a7434119ee7c00cad67515c8b24be0941e525647a0c993"} Jul 15 12:00:40.253383 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.253346 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:40.599253 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.599215 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:40.659820 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.659790 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6b49f799ff-mxqxz"] Jul 15 12:00:40.674811 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.674781 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.677340 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.677305 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 15 12:00:40.677791 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.677733 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 15 12:00:40.678559 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.678533 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 15 12:00:40.678906 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.678833 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6bzh9\"" Jul 15 12:00:40.679247 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.679226 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 15 12:00:40.679523 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.679502 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ak7kg8toefu5c\"" Jul 15 12:00:40.680329 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.679700 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 15 12:00:40.681768 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.681722 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b49f799ff-mxqxz"] Jul 15 12:00:40.688296 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.688261 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:40.758555 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.758524 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.758692 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.758571 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvfsg\" (UniqueName: \"kubernetes.io/projected/8ce62b12-ec66-45e2-82b2-cf9910477b1c-kube-api-access-dvfsg\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.758692 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.758636 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.758692 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.758659 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ce62b12-ec66-45e2-82b2-cf9910477b1c-metrics-client-ca\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.758840 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.758723 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-thanos-querier-tls\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.758840 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.758803 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.758909 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.758840 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.758909 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.758869 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-grpc-tls\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.812610 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.812555 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:40.859761 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.859681 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.859761 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.859726 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvfsg\" (UniqueName: \"kubernetes.io/projected/8ce62b12-ec66-45e2-82b2-cf9910477b1c-kube-api-access-dvfsg\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.859932 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.859869 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.859932 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.859905 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ce62b12-ec66-45e2-82b2-cf9910477b1c-metrics-client-ca\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.859932 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.859929 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-thanos-querier-tls\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.860094 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.860061 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.860225 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.860108 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.860225 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.860143 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-grpc-tls\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.860619 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.860599 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ce62b12-ec66-45e2-82b2-cf9910477b1c-metrics-client-ca\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.862659 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.862637 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.862923 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.862904 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.863016 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.862996 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.863111 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.863092 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.863165 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.863116 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-thanos-querier-tls\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.863286 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.863267 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8ce62b12-ec66-45e2-82b2-cf9910477b1c-secret-grpc-tls\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.870403 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.870382 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvfsg\" (UniqueName: \"kubernetes.io/projected/8ce62b12-ec66-45e2-82b2-cf9910477b1c-kube-api-access-dvfsg\") pod \"thanos-querier-6b49f799ff-mxqxz\" (UID: \"8ce62b12-ec66-45e2-82b2-cf9910477b1c\") " pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:40.890613 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.890579 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:40.988490 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:40.988464 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:41.165627 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.164121 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b49f799ff-mxqxz"] Jul 15 12:00:41.209717 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:00:41.209687 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce62b12_ec66_45e2_82b2_cf9910477b1c.slice/crio-3aeb4af73bc7a25a637f87514439d22bc9a3da6e79280126cccb52667e0c773c WatchSource:0}: Error finding container 3aeb4af73bc7a25a637f87514439d22bc9a3da6e79280126cccb52667e0c773c: Status 404 returned error can't find the container with id 3aeb4af73bc7a25a637f87514439d22bc9a3da6e79280126cccb52667e0c773c Jul 15 12:00:41.499996 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.499910 2570 generic.go:356] "Generic (PLEG): container finished" podID="279e04c2-80d5-4f93-83b0-fb8ab35223f0" containerID="c37fd8860835455e54089be5e22df2ca8e0f775388078ecb21190483964eb334" exitCode=0 Jul 15 12:00:41.500337 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.499993 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chq6t" event={"ID":"279e04c2-80d5-4f93-83b0-fb8ab35223f0","Type":"ContainerDied","Data":"c37fd8860835455e54089be5e22df2ca8e0f775388078ecb21190483964eb334"} Jul 15 12:00:41.502201 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.502180 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" event={"ID":"f9924fcc-57eb-421b-8255-4ee0fbb198cb","Type":"ContainerStarted","Data":"237089dab443146a9c86fc79d6c10af26a8425843f3f20165aa88ac0a2f5184e"} Jul 15 12:00:41.502285 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.502239 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" event={"ID":"f9924fcc-57eb-421b-8255-4ee0fbb198cb","Type":"ContainerStarted","Data":"41c05d9dd9b3b1bd4d1ea985d16e42f8c5e4a2d00855b9a270747dae995a70e5"} Jul 15 12:00:41.502285 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.502263 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" event={"ID":"f9924fcc-57eb-421b-8255-4ee0fbb198cb","Type":"ContainerStarted","Data":"0b4523ba3ddc20155a1e73a47992536dfaec14fdb742d29e0e9b598228f3bc87"} Jul 15 12:00:41.503377 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.503353 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" event={"ID":"8ce62b12-ec66-45e2-82b2-cf9910477b1c","Type":"ContainerStarted","Data":"3aeb4af73bc7a25a637f87514439d22bc9a3da6e79280126cccb52667e0c773c"} Jul 15 12:00:41.505500 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.505480 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" event={"ID":"54e8bb88-05f0-4569-968e-93f700263ce5","Type":"ContainerStarted","Data":"cf2c5805f6553ed15d0c93ef002cb938bc5bd8ad6a032e2584cda8ecff688cb9"} Jul 15 12:00:41.536018 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.535965 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-nwkhg" podStartSLOduration=2.362380615 podStartE2EDuration="4.535949607s" totalImagesPullingTime="2.173568992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:00:41.534566543 +0000 UTC m=+108.948094896" watchObservedRunningTime="2026-07-15 12:00:41.535949607 +0000 UTC m=+108.949477969" Jul 15 12:00:41.552838 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.551520 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-qwrhj" podStartSLOduration=2.009025231 podStartE2EDuration="4.551501717s" totalImagesPullingTime="2.542476486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:00:41.551288646 +0000 UTC m=+108.964816998" watchObservedRunningTime="2026-07-15 12:00:41.551501717 +0000 UTC m=+108.965030072" Jul 15 12:00:41.957157 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.957120 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-794764c799-2ffql"] Jul 15 12:00:41.970686 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.970654 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-794764c799-2ffql"] Jul 15 12:00:41.970920 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.970800 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:41.973698 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.973673 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 15 12:00:41.973853 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.973814 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 15 12:00:41.973933 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.973908 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fkqfommvtujo4\"" Jul 15 12:00:41.974093 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.974076 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-j76jm\"" Jul 15 12:00:41.974093 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.974089 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 15 12:00:41.974212 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:41.973679 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 15 12:00:42.071598 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.071560 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-secret-metrics-server-tls\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.071777 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.071614 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-secret-metrics-server-client-certs\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.071777 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.071684 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.071777 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.071727 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-metrics-server-audit-profiles\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.071945 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.071837 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb5vm\" (UniqueName: \"kubernetes.io/projected/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-kube-api-access-hb5vm\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.071945 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.071881 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-client-ca-bundle\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.071945 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.071904 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-audit-log\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.173373 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.173287 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-client-ca-bundle\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.173373 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.173334 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-audit-log\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.173594 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.173381 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-secret-metrics-server-tls\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.173594 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.173424 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-secret-metrics-server-client-certs\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.174380 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.173759 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.174380 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.173809 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-metrics-server-audit-profiles\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.174380 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.173890 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5vm\" (UniqueName: \"kubernetes.io/projected/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-kube-api-access-hb5vm\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.174380 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.173908 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-audit-log\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.174672 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.174472 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.175001 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.174765 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-metrics-server-audit-profiles\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.176462 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.176437 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-secret-metrics-server-client-certs\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.176913 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.176886 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-client-ca-bundle\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.177078 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.177055 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-secret-metrics-server-tls\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.182247 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.182222 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5vm\" (UniqueName: \"kubernetes.io/projected/028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3-kube-api-access-hb5vm\") pod \"metrics-server-794764c799-2ffql\" (UID: \"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3\") " pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.282913 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.282881 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:00:42.443050 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.442967 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78db486f6-mpwwc"] Jul 15 12:00:42.447162 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.447131 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-794764c799-2ffql"] Jul 15 12:00:42.450573 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:00:42.450544 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028f65a3_a2d2_4e26_a8ae_a1a2fe5027f3.slice/crio-4365a9809cc8f2e41a621d4da17f9bef8cf6a6805834411126c692d6694a5574 WatchSource:0}: Error finding container 4365a9809cc8f2e41a621d4da17f9bef8cf6a6805834411126c692d6694a5574: Status 404 returned error can't find the container with id 4365a9809cc8f2e41a621d4da17f9bef8cf6a6805834411126c692d6694a5574 Jul 15 12:00:42.459228 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.459208 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r"] Jul 15 12:00:42.462725 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.462706 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r" Jul 15 12:00:42.463579 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.463548 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:42.464639 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.464619 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-sztfn\"" Jul 15 12:00:42.464731 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.464619 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 15 12:00:42.475652 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.475585 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r"] Jul 15 12:00:42.512361 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.512317 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chq6t" event={"ID":"279e04c2-80d5-4f93-83b0-fb8ab35223f0","Type":"ContainerStarted","Data":"81ad3b1d671537adfc79f4d1b91d1d35754606e2288c554af245beb16e8f29f6"} Jul 15 12:00:42.512807 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.512376 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chq6t" event={"ID":"279e04c2-80d5-4f93-83b0-fb8ab35223f0","Type":"ContainerStarted","Data":"31e621e638e2552b5728891b06912ce080675ecdcc6467827947c31d9155bcb0"} Jul 15 12:00:42.513774 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.513725 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-794764c799-2ffql" event={"ID":"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3","Type":"ContainerStarted","Data":"4365a9809cc8f2e41a621d4da17f9bef8cf6a6805834411126c692d6694a5574"} Jul 15 12:00:42.537196 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.537133 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-chq6t" podStartSLOduration=3.154687393 podStartE2EDuration="5.53711673s" totalImagesPullingTime="2.282300064s" totalInitContainerRuntime="100.129273ms" isStatefulPod=true podCreationTimestamp="2026-07-15 12:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:00:42.536504293 +0000 UTC m=+109.950032646" watchObservedRunningTime="2026-07-15 12:00:42.53711673 +0000 UTC m=+109.950645081" Jul 15 12:00:42.577215 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.577173 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ll59\" (UniqueName: \"kubernetes.io/projected/3a397af2-2ff1-4296-b8e9-3c7b3888aa81-kube-api-access-8ll59\") pod \"monitoring-plugin-5f68df7f58-s5r7r\" (UID: \"3a397af2-2ff1-4296-b8e9-3c7b3888aa81\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r" Jul 15 12:00:42.577372 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.577228 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a397af2-2ff1-4296-b8e9-3c7b3888aa81-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-s5r7r\" (UID: \"3a397af2-2ff1-4296-b8e9-3c7b3888aa81\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r" Jul 15 12:00:42.677709 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.677677 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a397af2-2ff1-4296-b8e9-3c7b3888aa81-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-s5r7r\" (UID: \"3a397af2-2ff1-4296-b8e9-3c7b3888aa81\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r" Jul 15 12:00:42.677909 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:42.677877 2570 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 15 12:00:42.677978 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:00:42.677969 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a397af2-2ff1-4296-b8e9-3c7b3888aa81-monitoring-plugin-cert podName:3a397af2-2ff1-4296-b8e9-3c7b3888aa81 nodeName:}" failed. No retries permitted until 2026-07-15 12:00:43.177945621 +0000 UTC m=+110.591473967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/3a397af2-2ff1-4296-b8e9-3c7b3888aa81-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-s5r7r" (UID: "3a397af2-2ff1-4296-b8e9-3c7b3888aa81") : secret "monitoring-plugin-cert" not found Jul 15 12:00:42.678397 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.678279 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ll59\" (UniqueName: \"kubernetes.io/projected/3a397af2-2ff1-4296-b8e9-3c7b3888aa81-kube-api-access-8ll59\") pod \"monitoring-plugin-5f68df7f58-s5r7r\" (UID: \"3a397af2-2ff1-4296-b8e9-3c7b3888aa81\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r" Jul 15 12:00:42.687170 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.687148 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ll59\" (UniqueName: \"kubernetes.io/projected/3a397af2-2ff1-4296-b8e9-3c7b3888aa81-kube-api-access-8ll59\") pod \"monitoring-plugin-5f68df7f58-s5r7r\" (UID: \"3a397af2-2ff1-4296-b8e9-3c7b3888aa81\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r" Jul 15 12:00:42.794328 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.794239 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:42.969263 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:42.969220 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:43.182866 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.182825 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a397af2-2ff1-4296-b8e9-3c7b3888aa81-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-s5r7r\" (UID: \"3a397af2-2ff1-4296-b8e9-3c7b3888aa81\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r" Jul 15 12:00:43.185639 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.185605 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3a397af2-2ff1-4296-b8e9-3c7b3888aa81-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-s5r7r\" (UID: \"3a397af2-2ff1-4296-b8e9-3c7b3888aa81\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r" Jul 15 12:00:43.374671 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.374639 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r" Jul 15 12:00:43.507464 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.507431 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r"] Jul 15 12:00:43.510085 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:00:43.510057 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a397af2_2ff1_4296_b8e9_3c7b3888aa81.slice/crio-c23a2676791c8d8b6677832ec96ee0fc77bb285041d9d02ec35c7b001ff21593 WatchSource:0}: Error finding container c23a2676791c8d8b6677832ec96ee0fc77bb285041d9d02ec35c7b001ff21593: Status 404 returned error can't find the container with id c23a2676791c8d8b6677832ec96ee0fc77bb285041d9d02ec35c7b001ff21593 Jul 15 12:00:43.518717 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.518691 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r" event={"ID":"3a397af2-2ff1-4296-b8e9-3c7b3888aa81","Type":"ContainerStarted","Data":"c23a2676791c8d8b6677832ec96ee0fc77bb285041d9d02ec35c7b001ff21593"} Jul 15 12:00:43.520625 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.520600 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" event={"ID":"8ce62b12-ec66-45e2-82b2-cf9910477b1c","Type":"ContainerStarted","Data":"6aee62d2168509dcff4c37e49c2c5e5810e1728dc8c546b335d583817d27286e"} Jul 15 12:00:43.520711 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.520655 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" event={"ID":"8ce62b12-ec66-45e2-82b2-cf9910477b1c","Type":"ContainerStarted","Data":"2b9132da5949ed2c790e6ff81a15c2c857784de8f164e8527c3d46872c03231b"} Jul 15 12:00:43.520711 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.520682 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" event={"ID":"8ce62b12-ec66-45e2-82b2-cf9910477b1c","Type":"ContainerStarted","Data":"75c2fc118050f0de12c04ac6b0ea7c64ccd0bc8050c7b26dd068f466875bd3c9"} Jul 15 12:00:43.850779 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.850721 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 12:00:43.853780 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.853758 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.855969 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.855950 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 15 12:00:43.856140 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.856117 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9likdptcf6t6n\"" Jul 15 12:00:43.856198 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.856148 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 15 12:00:43.856455 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.856419 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 15 12:00:43.856583 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.856489 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 15 12:00:43.856583 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.856499 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-wd4ps\"" Jul 15 12:00:43.856583 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.856553 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 15 12:00:43.857089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.856792 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 15 12:00:43.857089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.856820 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 15 12:00:43.857089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.856835 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 15 12:00:43.857089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.856850 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 15 12:00:43.857089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.856880 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 15 12:00:43.857089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.856939 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 15 12:00:43.857089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.857012 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 15 12:00:43.857089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.857062 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 15 12:00:43.859625 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.859476 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 15 12:00:43.878444 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.878413 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 12:00:43.989342 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989305 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.989515 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989356 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.989515 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989388 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16cb1810-527b-4762-bb68-b35a23cbb19d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.989515 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989422 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.989515 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989446 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-config\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.989515 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989476 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.989515 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989496 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.989724 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989522 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16cb1810-527b-4762-bb68-b35a23cbb19d-config-out\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.989724 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989572 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.989724 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989605 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.989724 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989638 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.989724 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989655 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.989724 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989677 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.989724 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989700 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-web-config\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.989724 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989719 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tns5c\" (UniqueName: \"kubernetes.io/projected/16cb1810-527b-4762-bb68-b35a23cbb19d-kube-api-access-tns5c\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.990019 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989759 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.990019 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989773 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.990019 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989788 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16cb1810-527b-4762-bb68-b35a23cbb19d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.990019 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989805 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:43.990019 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:43.989819 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.090477 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.090450 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-web-config\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.090563 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.090494 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tns5c\" (UniqueName: \"kubernetes.io/projected/16cb1810-527b-4762-bb68-b35a23cbb19d-kube-api-access-tns5c\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.090563 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.090528 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.090721 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.090699 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.090813 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.090769 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16cb1810-527b-4762-bb68-b35a23cbb19d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.090813 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.090802 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.090919 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.090830 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.091005 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.090975 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.091052 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.091023 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.091089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.091063 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16cb1810-527b-4762-bb68-b35a23cbb19d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.091122 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.091099 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.091168 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.091130 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-config\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.091168 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.091149 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16cb1810-527b-4762-bb68-b35a23cbb19d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.091343 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.091310 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.092272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.091505 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.092272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.091542 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16cb1810-527b-4762-bb68-b35a23cbb19d-config-out\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.092272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.091602 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.092272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.091657 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.092272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.091705 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.092272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.091752 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.092272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.091794 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.092272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.092078 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.092730 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.092278 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.093807 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.093789 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.093954 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.093932 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.095837 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.094137 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.095837 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.094151 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-web-config\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.095837 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.094579 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.095837 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.095237 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.095837 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.095760 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.098041 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.096544 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.098160 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.098137 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16cb1810-527b-4762-bb68-b35a23cbb19d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.098502 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.098481 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16cb1810-527b-4762-bb68-b35a23cbb19d-config-out\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.098686 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.098661 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-config\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.099578 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.099553 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.100423 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.099654 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16cb1810-527b-4762-bb68-b35a23cbb19d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.100423 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.100253 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.101281 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.101234 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.101421 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.101389 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tns5c\" (UniqueName: \"kubernetes.io/projected/16cb1810-527b-4762-bb68-b35a23cbb19d-kube-api-access-tns5c\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.101724 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.101705 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16cb1810-527b-4762-bb68-b35a23cbb19d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"16cb1810-527b-4762-bb68-b35a23cbb19d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.116556 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.116523 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:44.173860 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.173833 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:44.358646 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.358077 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 12:00:44.475145 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.475108 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:44.525938 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.525897 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16cb1810-527b-4762-bb68-b35a23cbb19d","Type":"ContainerStarted","Data":"1c90f85da2d8b53db1b27f8f04f05ad489bd1aa19ac2847161f5fadc2f42b8b3"} Jul 15 12:00:44.529327 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.529295 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" event={"ID":"8ce62b12-ec66-45e2-82b2-cf9910477b1c","Type":"ContainerStarted","Data":"c774f85a52ebd26093f84b00cc1142453f23759ac080f69244e754fff9cc0c22"} Jul 15 12:00:44.529480 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.529363 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" event={"ID":"8ce62b12-ec66-45e2-82b2-cf9910477b1c","Type":"ContainerStarted","Data":"4c6e794111e2cc82d812a8b4465413faadea2348e8bbc6c9bdaeefbac8ba084a"} Jul 15 12:00:44.529480 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.529402 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" event={"ID":"8ce62b12-ec66-45e2-82b2-cf9910477b1c","Type":"ContainerStarted","Data":"7871b8a07e79aaea530c6f21e554b02b53737c21752d6bf96bc570625198a35a"} Jul 15 12:00:44.529581 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.529478 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:44.553795 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.553711 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" podStartSLOduration=1.92321348 podStartE2EDuration="4.553691114s" totalImagesPullingTime="2.630477634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:00:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:00:44.553573404 +0000 UTC m=+111.967101770" watchObservedRunningTime="2026-07-15 12:00:44.553691114 +0000 UTC m=+111.967219465" Jul 15 12:00:44.589518 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.589476 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:44.651619 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:44.651532 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:45.142800 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:45.142761 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:45.243345 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:45.243250 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:45.536428 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:45.536343 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r" event={"ID":"3a397af2-2ff1-4296-b8e9-3c7b3888aa81","Type":"ContainerStarted","Data":"8d657f75f3c9694b57d0a453f938daf499ae5ef9fe22861fdb63564895164fe5"} Jul 15 12:00:45.536889 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:45.536518 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r" Jul 15 12:00:45.542597 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:45.542573 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r" Jul 15 12:00:45.567886 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:45.567820 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-s5r7r" podStartSLOduration=2.132299173 podStartE2EDuration="3.567798058s" totalImagesPullingTime="1.435498885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:00:45.551992185 +0000 UTC m=+112.965520536" watchObservedRunningTime="2026-07-15 12:00:45.567798058 +0000 UTC m=+112.981326409" Jul 15 12:00:46.481172 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:46.481141 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-24s4z" Jul 15 12:00:46.541721 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:46.541679 2570 generic.go:356] "Generic (PLEG): container finished" podID="16cb1810-527b-4762-bb68-b35a23cbb19d" containerID="e0c95e8a2c332be475c64d74e8abb8310288724964f4c2799afd10696c65b5cc" exitCode=0 Jul 15 12:00:46.542192 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:46.541838 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16cb1810-527b-4762-bb68-b35a23cbb19d","Type":"ContainerDied","Data":"e0c95e8a2c332be475c64d74e8abb8310288724964f4c2799afd10696c65b5cc"} Jul 15 12:00:46.543720 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:46.543698 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-794764c799-2ffql" event={"ID":"028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3","Type":"ContainerStarted","Data":"4e769e91d6aa74ad9c3fb84ba426154d095e922ef95d77f5309ef612cd1330e8"} Jul 15 12:00:46.583442 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:46.583388 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-794764c799-2ffql" podStartSLOduration=2.009905726 podStartE2EDuration="5.58336793s" totalImagesPullingTime="3.573462204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:00:46.582898574 +0000 UTC m=+113.996426926" watchObservedRunningTime="2026-07-15 12:00:46.58336793 +0000 UTC m=+113.996896345" Jul 15 12:00:49.514835 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:49.514797 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:50.542531 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:50.542503 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6b49f799ff-mxqxz" Jul 15 12:00:50.566636 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:50.566603 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16cb1810-527b-4762-bb68-b35a23cbb19d","Type":"ContainerStarted","Data":"b423cecd4693aef34df87b39d8dcb1848bad6a2411ea27a3d6f054e2c4d9712c"} Jul 15 12:00:50.566775 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:50.566677 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16cb1810-527b-4762-bb68-b35a23cbb19d","Type":"ContainerStarted","Data":"473889b9e6d471822272fc28df2ab891ddb733b4cc795112b221fda052d8e952"} Jul 15 12:00:50.566775 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:50.566717 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16cb1810-527b-4762-bb68-b35a23cbb19d","Type":"ContainerStarted","Data":"540fdde99613100df451586b9036630dbcb6cbe7c43ff821224329b23a64ef6f"} Jul 15 12:00:50.566869 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:50.566791 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16cb1810-527b-4762-bb68-b35a23cbb19d","Type":"ContainerStarted","Data":"6264316d70bfd0c6f41bd8c44d21a1bd5f01ca1471f35a24f8fdbd2376f4b1c7"} Jul 15 12:00:50.566869 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:50.566838 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16cb1810-527b-4762-bb68-b35a23cbb19d","Type":"ContainerStarted","Data":"e2569dc31eb83feb4531c468753fd82c3999e3f81f2d4906bf8b258ea3dd9d8e"} Jul 15 12:00:50.566928 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:50.566876 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16cb1810-527b-4762-bb68-b35a23cbb19d","Type":"ContainerStarted","Data":"c17794bb3d9d6884944b27917d3b97fdfed0e5d0b30d20cc64ea498a8188deca"} Jul 15 12:00:50.598070 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:50.598000 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9960187339999997 podStartE2EDuration="7.597978852s" totalImagesPullingTime="4.577182281s" totalInitContainerRuntime="24.777837ms" isStatefulPod=false podCreationTimestamp="2026-07-15 12:00:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:00:50.595106236 +0000 UTC m=+118.008634790" watchObservedRunningTime="2026-07-15 12:00:50.597978852 +0000 UTC m=+118.011507205" Jul 15 12:00:52.834945 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:52.834907 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:52.894199 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:52.894166 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:53.915827 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:53.915782 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:54.015705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:54.015639 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:54.115445 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:54.115409 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:54.174538 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:54.174473 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:00:54.190394 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:54.190368 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:00:59.699870 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:00:59.699835 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:01.322194 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:01.322153 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:01.909355 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:01.909318 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:01.972871 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:01.972834 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:02.283677 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:02.283592 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:01:02.283677 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:02.283650 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:01:02.868823 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:02.868774 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs\") pod \"network-metrics-daemon-l4d6j\" (UID: \"76443aa4-c20f-4f32-8bbd-f88a776b0b34\") " pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 12:01:02.871320 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:02.871298 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76443aa4-c20f-4f32-8bbd-f88a776b0b34-metrics-certs\") pod \"network-metrics-daemon-l4d6j\" (UID: \"76443aa4-c20f-4f32-8bbd-f88a776b0b34\") " pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 12:01:02.898321 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:02.898291 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f876h\"" Jul 15 12:01:02.906951 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:02.906929 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l4d6j" Jul 15 12:01:02.986293 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:02.986262 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:03.082946 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:03.082913 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:03.410026 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:03.409997 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l4d6j"] Jul 15 12:01:03.414494 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:01:03.414466 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76443aa4_c20f_4f32_8bbd_f88a776b0b34.slice/crio-52481be4037072be4b590713676eb8a372b31ed0969148176ce2d8cf63e6ed78 WatchSource:0}: Error finding container 52481be4037072be4b590713676eb8a372b31ed0969148176ce2d8cf63e6ed78: Status 404 returned error can't find the container with id 52481be4037072be4b590713676eb8a372b31ed0969148176ce2d8cf63e6ed78 Jul 15 12:01:03.616977 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:03.616939 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l4d6j" event={"ID":"76443aa4-c20f-4f32-8bbd-f88a776b0b34","Type":"ContainerStarted","Data":"52481be4037072be4b590713676eb8a372b31ed0969148176ce2d8cf63e6ed78"} Jul 15 12:01:04.623417 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:04.623384 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l4d6j" event={"ID":"76443aa4-c20f-4f32-8bbd-f88a776b0b34","Type":"ContainerStarted","Data":"79621c1182a92de4ea5f764381c21fc79f9e590e8b225242fd6b3ad31fc8db6b"} Jul 15 12:01:05.630373 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:05.630340 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l4d6j" event={"ID":"76443aa4-c20f-4f32-8bbd-f88a776b0b34","Type":"ContainerStarted","Data":"7942063b42b43101ee86698a35eaaf280ecba6b458f43651167210ce9a3fb05a"} Jul 15 12:01:05.647650 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:05.647561 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l4d6j" podStartSLOduration=131.534337113 podStartE2EDuration="2m12.647542013s" totalImagesPullingTime="1.1132049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 11:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:01:05.646753463 +0000 UTC m=+133.060281814" watchObservedRunningTime="2026-07-15 12:01:05.647542013 +0000 UTC m=+133.061070367" Jul 15 12:01:05.779315 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:05.779273 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:05.872274 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:05.872239 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:05.972828 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:05.972722 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:06.048017 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:06.047982 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:07.465624 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.465583 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78db486f6-mpwwc" podUID="a8212cbc-ce8c-4c5b-9976-e76c68a70808" containerName="console" containerID="cri-o://640e0f05cf860ebd95f496286774cff83da2f9bdd5e53a69c14b91a4225a409d" gracePeriod=15 Jul 15 12:01:07.640239 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.640205 2570 generic.go:356] "Generic (PLEG): container finished" podID="a8212cbc-ce8c-4c5b-9976-e76c68a70808" containerID="640e0f05cf860ebd95f496286774cff83da2f9bdd5e53a69c14b91a4225a409d" exitCode=0 Jul 15 12:01:07.640363 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.640265 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78db486f6-mpwwc" event={"ID":"a8212cbc-ce8c-4c5b-9976-e76c68a70808","Type":"ContainerDied","Data":"640e0f05cf860ebd95f496286774cff83da2f9bdd5e53a69c14b91a4225a409d"} Jul 15 12:01:07.714687 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.714664 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:01:07.812498 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.812417 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-serving-cert\") pod \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " Jul 15 12:01:07.812498 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.812452 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-oauth-config\") pod \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " Jul 15 12:01:07.812498 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.812476 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-config\") pod \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " Jul 15 12:01:07.812498 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.812493 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-oauth-serving-cert\") pod \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " Jul 15 12:01:07.812836 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.812553 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-service-ca\") pod \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " Jul 15 12:01:07.812836 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.812601 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv5q9\" (UniqueName: \"kubernetes.io/projected/a8212cbc-ce8c-4c5b-9976-e76c68a70808-kube-api-access-kv5q9\") pod \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\" (UID: \"a8212cbc-ce8c-4c5b-9976-e76c68a70808\") " Jul 15 12:01:07.812995 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.812971 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-config" (OuterVolumeSpecName: "console-config") pod "a8212cbc-ce8c-4c5b-9976-e76c68a70808" (UID: "a8212cbc-ce8c-4c5b-9976-e76c68a70808"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:01:07.813035 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.812993 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a8212cbc-ce8c-4c5b-9976-e76c68a70808" (UID: "a8212cbc-ce8c-4c5b-9976-e76c68a70808"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:01:07.813035 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.812975 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-service-ca" (OuterVolumeSpecName: "service-ca") pod "a8212cbc-ce8c-4c5b-9976-e76c68a70808" (UID: "a8212cbc-ce8c-4c5b-9976-e76c68a70808"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:01:07.814921 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.814896 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a8212cbc-ce8c-4c5b-9976-e76c68a70808" (UID: "a8212cbc-ce8c-4c5b-9976-e76c68a70808"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:01:07.815043 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.815015 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a8212cbc-ce8c-4c5b-9976-e76c68a70808" (UID: "a8212cbc-ce8c-4c5b-9976-e76c68a70808"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:01:07.815136 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.815086 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8212cbc-ce8c-4c5b-9976-e76c68a70808-kube-api-access-kv5q9" (OuterVolumeSpecName: "kube-api-access-kv5q9") pod "a8212cbc-ce8c-4c5b-9976-e76c68a70808" (UID: "a8212cbc-ce8c-4c5b-9976-e76c68a70808"). InnerVolumeSpecName "kube-api-access-kv5q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:01:07.913350 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.913319 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-serving-cert\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:01:07.913350 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.913344 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-oauth-config\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:01:07.913350 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.913355 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-console-config\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:01:07.913564 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.913365 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-oauth-serving-cert\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:01:07.913564 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.913372 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8212cbc-ce8c-4c5b-9976-e76c68a70808-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:01:07.913564 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:07.913382 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv5q9\" (UniqueName: \"kubernetes.io/projected/a8212cbc-ce8c-4c5b-9976-e76c68a70808-kube-api-access-kv5q9\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:01:08.603097 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:08.603052 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:08.646170 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:08.646142 2570 generic.go:356] "Generic (PLEG): container finished" podID="74bcecd1-b162-4ce3-b731-d47b223860a8" containerID="1c081c401ee4351d37cddbd59c88f144d1acadc4dd342456902beb2d207d0d06" exitCode=0 Jul 15 12:01:08.646313 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:08.646218 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" event={"ID":"74bcecd1-b162-4ce3-b731-d47b223860a8","Type":"ContainerDied","Data":"1c081c401ee4351d37cddbd59c88f144d1acadc4dd342456902beb2d207d0d06"} Jul 15 12:01:08.646586 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:08.646569 2570 scope.go:117] "RemoveContainer" containerID="1c081c401ee4351d37cddbd59c88f144d1acadc4dd342456902beb2d207d0d06" Jul 15 12:01:08.647730 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:08.647715 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78db486f6-mpwwc" Jul 15 12:01:08.647730 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:08.647721 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78db486f6-mpwwc" event={"ID":"a8212cbc-ce8c-4c5b-9976-e76c68a70808","Type":"ContainerDied","Data":"118267eaf80de58aebc33f810fa0f248c33d6c809f9a139621dd5db13a52a55a"} Jul 15 12:01:08.647876 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:08.647782 2570 scope.go:117] "RemoveContainer" containerID="640e0f05cf860ebd95f496286774cff83da2f9bdd5e53a69c14b91a4225a409d" Jul 15 12:01:08.685356 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:08.685241 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78db486f6-mpwwc"] Jul 15 12:01:08.690462 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:08.690430 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:08.696393 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:08.696370 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78db486f6-mpwwc"] Jul 15 12:01:09.090708 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:09.090674 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8212cbc-ce8c-4c5b-9976-e76c68a70808" path="/var/lib/kubelet/pods/a8212cbc-ce8c-4c5b-9976-e76c68a70808/volumes" Jul 15 12:01:09.653916 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:09.653885 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-ftmvb" event={"ID":"74bcecd1-b162-4ce3-b731-d47b223860a8","Type":"ContainerStarted","Data":"1168323eb491a55d6c28f3f6f36e13c64682866426655c54f8e02c8208685371"} Jul 15 12:01:20.910799 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:20.910765 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:21.011512 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:21.011475 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:22.288396 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:22.288365 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:01:22.292367 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:22.292346 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-794764c799-2ffql" Jul 15 12:01:44.174167 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:44.174078 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:01:44.190306 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:44.190281 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:01:44.810381 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:44.810352 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:01:56.922343 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:56.922310 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:57.033831 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:57.033782 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:01:57.657669 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:01:57.657614 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:05.778006 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:05.777959 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:05.857341 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:05.857305 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:05.930350 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:05.930313 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:06.511123 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:06.511089 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:06.620899 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:06.620871 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:12.174427 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:12.174393 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:12.276154 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:12.276116 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:17.511063 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:17.511023 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:17.584126 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:17.584087 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:24.127931 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:24.127894 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:24.194486 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:24.194456 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:27.304863 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:27.304819 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:27.404934 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:27.404892 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:27.505802 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:27.505765 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:29.950359 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:29.950314 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:32.284405 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:32.284368 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:32.380330 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:32.380295 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:44.492544 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:44.492503 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:02:44.581138 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:02:44.581105 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:03:27.199017 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:03:27.198972 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:03:30.752480 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:03:30.752446 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:03:30.852566 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:03:30.852518 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:03:53.024942 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:03:53.024914 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:03:53.030910 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:03:53.030881 2570 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 12:03:53.037541 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:03:53.037520 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:03:57.632944 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:03:57.632895 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:03:58.995007 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:03:58.994968 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:01.301477 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:01.301441 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:01.397950 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:01.397915 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:05.693495 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:05.693454 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:05.789263 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:05.789222 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:05.908168 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:05.908129 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:06.063476 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:06.063389 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:06.277195 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:06.277138 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:06.378107 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:06.378069 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:07.913006 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:07.912964 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:08.118622 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:08.118583 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:16.107480 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:16.107433 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:34.593312 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:34.593272 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:36.655152 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:36.655123 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:40.400895 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:40.400859 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:40.503382 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:40.503352 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:42.332152 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:42.332113 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:43.589388 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:43.589334 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:43.960934 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:43.960900 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:48.498485 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.498453 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:48.553593 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.553561 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c"] Jul 15 12:04:48.553903 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.553889 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8212cbc-ce8c-4c5b-9976-e76c68a70808" containerName="console" Jul 15 12:04:48.553943 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.553905 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8212cbc-ce8c-4c5b-9976-e76c68a70808" containerName="console" Jul 15 12:04:48.553984 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.553957 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8212cbc-ce8c-4c5b-9976-e76c68a70808" containerName="console" Jul 15 12:04:48.556578 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.556559 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:04:48.565155 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.565136 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 12:04:48.565264 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.565188 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 12:04:48.565341 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.565327 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 12:04:48.565469 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.565450 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fdwgb\"" Jul 15 12:04:48.565569 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.565551 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 12:04:48.565613 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.565568 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 12:04:48.594279 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.594247 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c"] Jul 15 12:04:48.608777 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.608726 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:48.649065 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.649035 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cc7e65d9-a799-4475-b9b4-450d0c2627ce-manager-config\") pod \"lws-controller-manager-5dc89645fd-k9j5c\" (UID: \"cc7e65d9-a799-4475-b9b4-450d0c2627ce\") " pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:04:48.649217 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.649075 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc7e65d9-a799-4475-b9b4-450d0c2627ce-cert\") pod \"lws-controller-manager-5dc89645fd-k9j5c\" (UID: \"cc7e65d9-a799-4475-b9b4-450d0c2627ce\") " pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:04:48.649217 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.649161 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc7e65d9-a799-4475-b9b4-450d0c2627ce-metrics-cert\") pod \"lws-controller-manager-5dc89645fd-k9j5c\" (UID: \"cc7e65d9-a799-4475-b9b4-450d0c2627ce\") " pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:04:48.649217 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.649185 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ndzt\" (UniqueName: \"kubernetes.io/projected/cc7e65d9-a799-4475-b9b4-450d0c2627ce-kube-api-access-8ndzt\") pod \"lws-controller-manager-5dc89645fd-k9j5c\" (UID: \"cc7e65d9-a799-4475-b9b4-450d0c2627ce\") " pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:04:48.750645 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.750538 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cc7e65d9-a799-4475-b9b4-450d0c2627ce-manager-config\") pod \"lws-controller-manager-5dc89645fd-k9j5c\" (UID: \"cc7e65d9-a799-4475-b9b4-450d0c2627ce\") " pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:04:48.750645 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.750595 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc7e65d9-a799-4475-b9b4-450d0c2627ce-cert\") pod \"lws-controller-manager-5dc89645fd-k9j5c\" (UID: \"cc7e65d9-a799-4475-b9b4-450d0c2627ce\") " pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:04:48.750645 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.750638 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc7e65d9-a799-4475-b9b4-450d0c2627ce-metrics-cert\") pod \"lws-controller-manager-5dc89645fd-k9j5c\" (UID: \"cc7e65d9-a799-4475-b9b4-450d0c2627ce\") " pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:04:48.750967 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.750654 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ndzt\" (UniqueName: \"kubernetes.io/projected/cc7e65d9-a799-4475-b9b4-450d0c2627ce-kube-api-access-8ndzt\") pod \"lws-controller-manager-5dc89645fd-k9j5c\" (UID: \"cc7e65d9-a799-4475-b9b4-450d0c2627ce\") " pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:04:48.751385 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.751361 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cc7e65d9-a799-4475-b9b4-450d0c2627ce-manager-config\") pod \"lws-controller-manager-5dc89645fd-k9j5c\" (UID: \"cc7e65d9-a799-4475-b9b4-450d0c2627ce\") " pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:04:48.753227 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.753199 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cc7e65d9-a799-4475-b9b4-450d0c2627ce-metrics-cert\") pod \"lws-controller-manager-5dc89645fd-k9j5c\" (UID: \"cc7e65d9-a799-4475-b9b4-450d0c2627ce\") " pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:04:48.753375 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.753354 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc7e65d9-a799-4475-b9b4-450d0c2627ce-cert\") pod \"lws-controller-manager-5dc89645fd-k9j5c\" (UID: \"cc7e65d9-a799-4475-b9b4-450d0c2627ce\") " pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:04:48.759104 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.759080 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ndzt\" (UniqueName: \"kubernetes.io/projected/cc7e65d9-a799-4475-b9b4-450d0c2627ce-kube-api-access-8ndzt\") pod \"lws-controller-manager-5dc89645fd-k9j5c\" (UID: \"cc7e65d9-a799-4475-b9b4-450d0c2627ce\") " pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:04:48.865681 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:48.865651 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:04:49.001529 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:49.001497 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c"] Jul 15 12:04:49.004020 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:04:49.003994 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7e65d9_a799_4475_b9b4_450d0c2627ce.slice/crio-226ed68335f896e90d7196a92b8187be7572ca46e33e84d1b9a6e2e08a9a9b79 WatchSource:0}: Error finding container 226ed68335f896e90d7196a92b8187be7572ca46e33e84d1b9a6e2e08a9a9b79: Status 404 returned error can't find the container with id 226ed68335f896e90d7196a92b8187be7572ca46e33e84d1b9a6e2e08a9a9b79 Jul 15 12:04:49.005975 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:49.005957 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:04:49.508436 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:49.508396 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" event={"ID":"cc7e65d9-a799-4475-b9b4-450d0c2627ce","Type":"ContainerStarted","Data":"226ed68335f896e90d7196a92b8187be7572ca46e33e84d1b9a6e2e08a9a9b79"} Jul 15 12:04:50.587058 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:50.587005 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:52.523242 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:52.523194 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" event={"ID":"cc7e65d9-a799-4475-b9b4-450d0c2627ce","Type":"ContainerStarted","Data":"84ec0a4331f7bb4ee007ca0a7e7417c9c3e1fb49ea8bbd830315e9076d02625e"} Jul 15 12:04:52.523671 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:52.523284 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:04:52.557580 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:52.557491 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" podStartSLOduration=1.899103845 podStartE2EDuration="4.557470135s" totalImagesPullingTime="2.65836629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:04:52.556216461 +0000 UTC m=+359.969744812" watchObservedRunningTime="2026-07-15 12:04:52.557470135 +0000 UTC m=+359.970998487" Jul 15 12:04:57.145307 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:57.145260 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:57.241862 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:57.241828 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:57.802019 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:57.801983 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:58.082991 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.082901 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:58.179031 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.178995 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:58.267268 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.267237 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f"] Jul 15 12:04:58.270916 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.270892 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.273454 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.273433 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 12:04:58.273590 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.273500 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 12:04:58.273641 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.273621 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 12:04:58.273802 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.273784 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 12:04:58.274109 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.274091 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vrmsp\"" Jul 15 12:04:58.288201 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.288165 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f"] Jul 15 12:04:58.396183 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.396148 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:58.442357 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.442318 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69ea5e3a-62a0-476b-80de-32af2b08ff93-tmp\") pod \"opendatahub-operator-controller-manager-694c4fcbc8-6vv9f\" (UID: \"69ea5e3a-62a0-476b-80de-32af2b08ff93\") " pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.442357 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.442356 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4d7b\" (UniqueName: \"kubernetes.io/projected/69ea5e3a-62a0-476b-80de-32af2b08ff93-kube-api-access-n4d7b\") pod \"opendatahub-operator-controller-manager-694c4fcbc8-6vv9f\" (UID: \"69ea5e3a-62a0-476b-80de-32af2b08ff93\") " pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.442544 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.442453 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69ea5e3a-62a0-476b-80de-32af2b08ff93-webhook-cert\") pod \"opendatahub-operator-controller-manager-694c4fcbc8-6vv9f\" (UID: \"69ea5e3a-62a0-476b-80de-32af2b08ff93\") " pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.442544 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.442493 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/69ea5e3a-62a0-476b-80de-32af2b08ff93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694c4fcbc8-6vv9f\" (UID: \"69ea5e3a-62a0-476b-80de-32af2b08ff93\") " pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.442544 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.442516 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/69ea5e3a-62a0-476b-80de-32af2b08ff93-manifests\") pod \"opendatahub-operator-controller-manager-694c4fcbc8-6vv9f\" (UID: \"69ea5e3a-62a0-476b-80de-32af2b08ff93\") " pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.495441 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.495402 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:58.542976 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.542941 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69ea5e3a-62a0-476b-80de-32af2b08ff93-tmp\") pod \"opendatahub-operator-controller-manager-694c4fcbc8-6vv9f\" (UID: \"69ea5e3a-62a0-476b-80de-32af2b08ff93\") " pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.542976 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.542976 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4d7b\" (UniqueName: \"kubernetes.io/projected/69ea5e3a-62a0-476b-80de-32af2b08ff93-kube-api-access-n4d7b\") pod \"opendatahub-operator-controller-manager-694c4fcbc8-6vv9f\" (UID: \"69ea5e3a-62a0-476b-80de-32af2b08ff93\") " pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.543174 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.543027 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69ea5e3a-62a0-476b-80de-32af2b08ff93-webhook-cert\") pod \"opendatahub-operator-controller-manager-694c4fcbc8-6vv9f\" (UID: \"69ea5e3a-62a0-476b-80de-32af2b08ff93\") " pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.543174 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.543045 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/69ea5e3a-62a0-476b-80de-32af2b08ff93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694c4fcbc8-6vv9f\" (UID: \"69ea5e3a-62a0-476b-80de-32af2b08ff93\") " pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.543174 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.543062 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/69ea5e3a-62a0-476b-80de-32af2b08ff93-manifests\") pod \"opendatahub-operator-controller-manager-694c4fcbc8-6vv9f\" (UID: \"69ea5e3a-62a0-476b-80de-32af2b08ff93\") " pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.543500 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.543476 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/69ea5e3a-62a0-476b-80de-32af2b08ff93-manifests\") pod \"opendatahub-operator-controller-manager-694c4fcbc8-6vv9f\" (UID: \"69ea5e3a-62a0-476b-80de-32af2b08ff93\") " pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.545593 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.545565 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69ea5e3a-62a0-476b-80de-32af2b08ff93-tmp\") pod \"opendatahub-operator-controller-manager-694c4fcbc8-6vv9f\" (UID: \"69ea5e3a-62a0-476b-80de-32af2b08ff93\") " pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.545708 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.545672 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/69ea5e3a-62a0-476b-80de-32af2b08ff93-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694c4fcbc8-6vv9f\" (UID: \"69ea5e3a-62a0-476b-80de-32af2b08ff93\") " pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.545975 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.545951 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69ea5e3a-62a0-476b-80de-32af2b08ff93-webhook-cert\") pod \"opendatahub-operator-controller-manager-694c4fcbc8-6vv9f\" (UID: \"69ea5e3a-62a0-476b-80de-32af2b08ff93\") " pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.552160 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.552137 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4d7b\" (UniqueName: \"kubernetes.io/projected/69ea5e3a-62a0-476b-80de-32af2b08ff93-kube-api-access-n4d7b\") pod \"opendatahub-operator-controller-manager-694c4fcbc8-6vv9f\" (UID: \"69ea5e3a-62a0-476b-80de-32af2b08ff93\") " pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.582105 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.582064 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:04:58.594031 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.593998 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:58.701355 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.701318 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:58.725949 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.725925 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f"] Jul 15 12:04:58.728774 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:04:58.728732 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ea5e3a_62a0_476b_80de_32af2b08ff93.slice/crio-73e14ceceb2b842fce78496f13779d600ad80d8c5ea8b67d30e3553a86d5c0a2 WatchSource:0}: Error finding container 73e14ceceb2b842fce78496f13779d600ad80d8c5ea8b67d30e3553a86d5c0a2: Status 404 returned error can't find the container with id 73e14ceceb2b842fce78496f13779d600ad80d8c5ea8b67d30e3553a86d5c0a2 Jul 15 12:04:58.802312 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.802271 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:58.893711 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.893675 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:58.995012 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:58.994908 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:04:59.555466 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:04:59.555419 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" event={"ID":"69ea5e3a-62a0-476b-80de-32af2b08ff93","Type":"ContainerStarted","Data":"73e14ceceb2b842fce78496f13779d600ad80d8c5ea8b67d30e3553a86d5c0a2"} Jul 15 12:05:03.530378 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:03.530343 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5dc89645fd-k9j5c" Jul 15 12:05:03.576209 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:03.576161 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" event={"ID":"69ea5e3a-62a0-476b-80de-32af2b08ff93","Type":"ContainerStarted","Data":"d90d6cd72b6f7852cf2695aa96225815b2590aabdf52247d0979e7e3222f54a7"} Jul 15 12:05:04.582396 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:04.582353 2570 generic.go:356] "Generic (PLEG): container finished" podID="69ea5e3a-62a0-476b-80de-32af2b08ff93" containerID="d90d6cd72b6f7852cf2695aa96225815b2590aabdf52247d0979e7e3222f54a7" exitCode=0 Jul 15 12:05:04.582812 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:04.582438 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" event={"ID":"69ea5e3a-62a0-476b-80de-32af2b08ff93","Type":"ContainerDied","Data":"d90d6cd72b6f7852cf2695aa96225815b2590aabdf52247d0979e7e3222f54a7"} Jul 15 12:05:06.592576 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:06.592480 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" event={"ID":"69ea5e3a-62a0-476b-80de-32af2b08ff93","Type":"ContainerStarted","Data":"92f7a5e4ee8b5f473ac1b2bad436d067bc5b7485cb3008390705f946f97a6b1b"} Jul 15 12:05:06.592576 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:06.592560 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:05:06.614860 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:06.614806 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" podStartSLOduration=2.196907179 podStartE2EDuration="8.614789861s" totalImagesPullingTime="5.335740086s" totalInitContainerRuntime="1.082142596s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:04:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:05:06.612576493 +0000 UTC m=+374.026104857" watchObservedRunningTime="2026-07-15 12:05:06.614789861 +0000 UTC m=+374.028318212" Jul 15 12:05:17.599139 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:17.599109 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-694c4fcbc8-6vv9f" Jul 15 12:05:19.191461 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:19.191431 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:19.331110 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:19.331067 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:19.782122 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:19.782088 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:19.933938 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:19.933907 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz"] Jul 15 12:05:19.941420 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:19.941402 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz" Jul 15 12:05:19.945121 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:19.945099 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-g5tqd\"" Jul 15 12:05:19.946235 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:19.946216 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 12:05:19.947381 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:19.947363 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 12:05:19.957909 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:19.957867 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz"] Jul 15 12:05:20.028077 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.028045 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/621592bf-b388-465a-843a-2c803e0c6c1c-tmp\") pod \"kube-auth-proxy-7d7f886b65-v8fmz\" (UID: \"621592bf-b388-465a-843a-2c803e0c6c1c\") " pod="openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz" Jul 15 12:05:20.028077 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.028080 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvrkv\" (UniqueName: \"kubernetes.io/projected/621592bf-b388-465a-843a-2c803e0c6c1c-kube-api-access-mvrkv\") pod \"kube-auth-proxy-7d7f886b65-v8fmz\" (UID: \"621592bf-b388-465a-843a-2c803e0c6c1c\") " pod="openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz" Jul 15 12:05:20.028291 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.028102 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/621592bf-b388-465a-843a-2c803e0c6c1c-tls-certs\") pod \"kube-auth-proxy-7d7f886b65-v8fmz\" (UID: \"621592bf-b388-465a-843a-2c803e0c6c1c\") " pod="openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz" Jul 15 12:05:20.129482 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.128888 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/621592bf-b388-465a-843a-2c803e0c6c1c-tmp\") pod \"kube-auth-proxy-7d7f886b65-v8fmz\" (UID: \"621592bf-b388-465a-843a-2c803e0c6c1c\") " pod="openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz" Jul 15 12:05:20.129482 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.128941 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvrkv\" (UniqueName: \"kubernetes.io/projected/621592bf-b388-465a-843a-2c803e0c6c1c-kube-api-access-mvrkv\") pod \"kube-auth-proxy-7d7f886b65-v8fmz\" (UID: \"621592bf-b388-465a-843a-2c803e0c6c1c\") " pod="openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz" Jul 15 12:05:20.129482 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.128977 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/621592bf-b388-465a-843a-2c803e0c6c1c-tls-certs\") pod \"kube-auth-proxy-7d7f886b65-v8fmz\" (UID: \"621592bf-b388-465a-843a-2c803e0c6c1c\") " pod="openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz" Jul 15 12:05:20.134925 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.132864 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/621592bf-b388-465a-843a-2c803e0c6c1c-tmp\") pod \"kube-auth-proxy-7d7f886b65-v8fmz\" (UID: \"621592bf-b388-465a-843a-2c803e0c6c1c\") " pod="openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz" Jul 15 12:05:20.136106 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.136056 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/621592bf-b388-465a-843a-2c803e0c6c1c-tls-certs\") pod \"kube-auth-proxy-7d7f886b65-v8fmz\" (UID: \"621592bf-b388-465a-843a-2c803e0c6c1c\") " pod="openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz" Jul 15 12:05:20.144553 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.144521 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvrkv\" (UniqueName: \"kubernetes.io/projected/621592bf-b388-465a-843a-2c803e0c6c1c-kube-api-access-mvrkv\") pod \"kube-auth-proxy-7d7f886b65-v8fmz\" (UID: \"621592bf-b388-465a-843a-2c803e0c6c1c\") " pod="openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz" Jul 15 12:05:20.177073 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.177039 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:20.250768 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.250482 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz" Jul 15 12:05:20.273825 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.273784 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:20.376166 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.376133 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:20.398097 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.398074 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz"] Jul 15 12:05:20.400634 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:05:20.400614 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621592bf_b388_465a_843a_2c803e0c6c1c.slice/crio-e6acd8f6ae9e1e0192079f95f7d56a6884c27529791f454ce58824567ebbdbe3 WatchSource:0}: Error finding container e6acd8f6ae9e1e0192079f95f7d56a6884c27529791f454ce58824567ebbdbe3: Status 404 returned error can't find the container with id e6acd8f6ae9e1e0192079f95f7d56a6884c27529791f454ce58824567ebbdbe3 Jul 15 12:05:20.495062 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.495021 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:20.658549 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.658466 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz" event={"ID":"621592bf-b388-465a-843a-2c803e0c6c1c","Type":"ContainerStarted","Data":"e6acd8f6ae9e1e0192079f95f7d56a6884c27529791f454ce58824567ebbdbe3"} Jul 15 12:05:20.999142 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:20.999052 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:21.202982 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:21.202902 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:21.326144 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:21.326044 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:21.385999 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:21.385964 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:22.420280 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:22.420224 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:23.674286 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:23.674255 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz" event={"ID":"621592bf-b388-465a-843a-2c803e0c6c1c","Type":"ContainerStarted","Data":"e8496fefd221d3455bd473f20fbf49de624954b4200a6ca40fa74c05d5b113a3"} Jul 15 12:05:23.694583 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:23.694530 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d7f886b65-v8fmz" podStartSLOduration=1.5634982 podStartE2EDuration="4.694511659s" totalImagesPullingTime="3.131013459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:05:23.692544756 +0000 UTC m=+391.106073108" watchObservedRunningTime="2026-07-15 12:05:23.694511659 +0000 UTC m=+391.108040011" Jul 15 12:05:27.150599 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:27.150561 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:27.250463 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:27.250425 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:27.549389 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:27.549311 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:27.646628 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:27.646590 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:27.819834 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:27.819705 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:31.100061 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:31.100025 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:31.184934 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:31.184898 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:31.259907 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:31.259865 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:31.382571 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:31.382479 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:31.558926 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:31.558886 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:32.416337 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:32.416297 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:32.518668 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:32.518628 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:32.642218 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:32.642184 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:36.938540 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:36.938502 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:05:36.999894 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:05:36.999866 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:06:08.344151 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:06:08.344117 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:06:08.447908 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:06:08.447864 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:06:20.801461 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:06:20.801380 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:06:20.922989 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:06:20.922945 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:06:21.025151 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:06:21.025110 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:06:49.227845 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:06:49.227802 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:06:49.835290 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:06:49.835250 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:06:50.847628 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:06:50.847592 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:06:50.948075 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:06:50.948044 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:06:54.232953 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:06:54.232921 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:06:57.017581 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:06:57.017544 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:06:57.287021 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:06:57.286929 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:06:59.555130 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:06:59.555065 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:06:59.657041 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:06:59.657005 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:02.544251 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:02.544209 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:04.647578 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:04.647540 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:04.749453 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:04.749415 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:04.850815 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:04.850775 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:04.950163 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:04.950086 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:08.429070 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:08.429024 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:08.500893 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:08.500847 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:12.651482 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:12.651447 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:12.756342 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:12.756305 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:13.571409 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.571372 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl"] Jul 15 12:07:13.574614 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.574596 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl" Jul 15 12:07:13.576658 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.576626 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 15 12:07:13.576792 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.576662 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 12:07:13.576865 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.576788 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 12:07:13.576865 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.576812 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 15 12:07:13.576987 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.576970 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vs7gg\"" Jul 15 12:07:13.583467 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.583441 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl"] Jul 15 12:07:13.584425 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.584405 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:13.709273 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.709243 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d95a5f8a-9a7c-414b-a526-6db1b266c278-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zgfdl\" (UID: \"d95a5f8a-9a7c-414b-a526-6db1b266c278\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl" Jul 15 12:07:13.709641 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.709286 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28qfc\" (UniqueName: \"kubernetes.io/projected/d95a5f8a-9a7c-414b-a526-6db1b266c278-kube-api-access-28qfc\") pod \"kuadrant-console-plugin-66b5774985-zgfdl\" (UID: \"d95a5f8a-9a7c-414b-a526-6db1b266c278\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl" Jul 15 12:07:13.709641 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.709313 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d95a5f8a-9a7c-414b-a526-6db1b266c278-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zgfdl\" (UID: \"d95a5f8a-9a7c-414b-a526-6db1b266c278\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl" Jul 15 12:07:13.806792 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.806761 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:13.809773 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.809727 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d95a5f8a-9a7c-414b-a526-6db1b266c278-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zgfdl\" (UID: \"d95a5f8a-9a7c-414b-a526-6db1b266c278\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl" Jul 15 12:07:13.809893 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.809819 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28qfc\" (UniqueName: \"kubernetes.io/projected/d95a5f8a-9a7c-414b-a526-6db1b266c278-kube-api-access-28qfc\") pod \"kuadrant-console-plugin-66b5774985-zgfdl\" (UID: \"d95a5f8a-9a7c-414b-a526-6db1b266c278\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl" Jul 15 12:07:13.809893 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.809865 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d95a5f8a-9a7c-414b-a526-6db1b266c278-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zgfdl\" (UID: \"d95a5f8a-9a7c-414b-a526-6db1b266c278\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl" Jul 15 12:07:13.810290 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.810271 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d95a5f8a-9a7c-414b-a526-6db1b266c278-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-zgfdl\" (UID: \"d95a5f8a-9a7c-414b-a526-6db1b266c278\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl" Jul 15 12:07:13.812317 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.812296 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d95a5f8a-9a7c-414b-a526-6db1b266c278-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-zgfdl\" (UID: \"d95a5f8a-9a7c-414b-a526-6db1b266c278\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl" Jul 15 12:07:13.816956 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.816934 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28qfc\" (UniqueName: \"kubernetes.io/projected/d95a5f8a-9a7c-414b-a526-6db1b266c278-kube-api-access-28qfc\") pod \"kuadrant-console-plugin-66b5774985-zgfdl\" (UID: \"d95a5f8a-9a7c-414b-a526-6db1b266c278\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl" Jul 15 12:07:13.883915 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:13.883881 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl" Jul 15 12:07:14.018072 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.018039 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl"] Jul 15 12:07:14.021539 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:07:14.021509 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd95a5f8a_9a7c_414b_a526_6db1b266c278.slice/crio-23feb3cb81531bf54c66e64688f7619538a9cd5f5765364adb8bc38c9915cf54 WatchSource:0}: Error finding container 23feb3cb81531bf54c66e64688f7619538a9cd5f5765364adb8bc38c9915cf54: Status 404 returned error can't find the container with id 23feb3cb81531bf54c66e64688f7619538a9cd5f5765364adb8bc38c9915cf54 Jul 15 12:07:14.140870 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.140777 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl" event={"ID":"d95a5f8a-9a7c-414b-a526-6db1b266c278","Type":"ContainerStarted","Data":"23feb3cb81531bf54c66e64688f7619538a9cd5f5765364adb8bc38c9915cf54"} Jul 15 12:07:14.382987 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.382951 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ff58cf98f-rklgw"] Jul 15 12:07:14.387151 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.387130 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.388966 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.388946 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 12:07:14.389777 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.389726 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 12:07:14.389887 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.389805 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 12:07:14.389887 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.389812 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 12:07:14.389887 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.389754 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-p4glj\"" Jul 15 12:07:14.389887 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.389851 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 12:07:14.390065 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.390032 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 12:07:14.390140 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.390124 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 12:07:14.395768 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.395696 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 12:07:14.396448 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.396426 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff58cf98f-rklgw"] Jul 15 12:07:14.442588 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.442552 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:14.514941 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.514907 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5e48be8-50e8-4b40-946e-60830b2369cd-console-serving-cert\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.515119 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.514958 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5e48be8-50e8-4b40-946e-60830b2369cd-oauth-serving-cert\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.515119 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.515020 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e48be8-50e8-4b40-946e-60830b2369cd-trusted-ca-bundle\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.515119 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.515053 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqrm2\" (UniqueName: \"kubernetes.io/projected/d5e48be8-50e8-4b40-946e-60830b2369cd-kube-api-access-bqrm2\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.515119 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.515072 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e48be8-50e8-4b40-946e-60830b2369cd-service-ca\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.515286 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.515121 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5e48be8-50e8-4b40-946e-60830b2369cd-console-oauth-config\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.515286 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.515172 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5e48be8-50e8-4b40-946e-60830b2369cd-console-config\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.542563 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.542521 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:14.615805 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.615768 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5e48be8-50e8-4b40-946e-60830b2369cd-console-oauth-config\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.615805 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.615807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5e48be8-50e8-4b40-946e-60830b2369cd-console-config\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.616034 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.615845 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5e48be8-50e8-4b40-946e-60830b2369cd-console-serving-cert\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.616034 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.615880 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5e48be8-50e8-4b40-946e-60830b2369cd-oauth-serving-cert\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.616034 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.615919 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e48be8-50e8-4b40-946e-60830b2369cd-trusted-ca-bundle\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.616034 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.615972 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqrm2\" (UniqueName: \"kubernetes.io/projected/d5e48be8-50e8-4b40-946e-60830b2369cd-kube-api-access-bqrm2\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.616034 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.615993 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e48be8-50e8-4b40-946e-60830b2369cd-service-ca\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.616634 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.616599 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5e48be8-50e8-4b40-946e-60830b2369cd-console-config\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.616770 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.616691 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e48be8-50e8-4b40-946e-60830b2369cd-service-ca\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.616826 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.616772 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5e48be8-50e8-4b40-946e-60830b2369cd-oauth-serving-cert\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.616900 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.616885 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5e48be8-50e8-4b40-946e-60830b2369cd-trusted-ca-bundle\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.619029 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.619006 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5e48be8-50e8-4b40-946e-60830b2369cd-console-serving-cert\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.619029 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.619026 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5e48be8-50e8-4b40-946e-60830b2369cd-console-oauth-config\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.624317 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.624296 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqrm2\" (UniqueName: \"kubernetes.io/projected/d5e48be8-50e8-4b40-946e-60830b2369cd-kube-api-access-bqrm2\") pod \"console-ff58cf98f-rklgw\" (UID: \"d5e48be8-50e8-4b40-946e-60830b2369cd\") " pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.698417 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.698335 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:14.828791 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:14.828753 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff58cf98f-rklgw"] Jul 15 12:07:14.831969 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:07:14.831939 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e48be8_50e8_4b40_946e_60830b2369cd.slice/crio-1d5cd49694aa04ce04789a2b4647ee33b6c94212f33c1bc0780930fe7d07badd WatchSource:0}: Error finding container 1d5cd49694aa04ce04789a2b4647ee33b6c94212f33c1bc0780930fe7d07badd: Status 404 returned error can't find the container with id 1d5cd49694aa04ce04789a2b4647ee33b6c94212f33c1bc0780930fe7d07badd Jul 15 12:07:15.146312 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:15.146275 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff58cf98f-rklgw" event={"ID":"d5e48be8-50e8-4b40-946e-60830b2369cd","Type":"ContainerStarted","Data":"4468eaac9bf6f61b92479b39ef2433fe8cb126a29719ec70ed0a2fb5bba3d8b9"} Jul 15 12:07:15.146479 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:15.146331 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff58cf98f-rklgw" event={"ID":"d5e48be8-50e8-4b40-946e-60830b2369cd","Type":"ContainerStarted","Data":"1d5cd49694aa04ce04789a2b4647ee33b6c94212f33c1bc0780930fe7d07badd"} Jul 15 12:07:15.167412 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:15.167361 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ff58cf98f-rklgw" podStartSLOduration=1.167342645 podStartE2EDuration="1.167342645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:07:15.165798075 +0000 UTC m=+502.579326423" watchObservedRunningTime="2026-07-15 12:07:15.167342645 +0000 UTC m=+502.580870996" Jul 15 12:07:16.154189 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:16.154137 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:16.255155 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:16.255117 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:24.003491 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:24.003450 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:24.105270 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:24.105223 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:24.600042 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:24.599964 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:24.699370 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:24.699312 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:24.699619 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:24.699419 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:24.705802 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:24.705772 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:24.908395 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:24.908305 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:25.199217 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:25.199123 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ff58cf98f-rklgw" Jul 15 12:07:25.378367 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:25.378328 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:25.481581 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:25.481483 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:25.585018 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:25.584936 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:27.006902 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:27.006858 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:27.107895 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:27.107862 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:29.079438 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:29.079395 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:32.195070 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:32.195031 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:32.296696 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:32.296662 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:34.772374 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:34.772324 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:34.875104 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:34.875064 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:39.264995 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:39.264953 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl" event={"ID":"d95a5f8a-9a7c-414b-a526-6db1b266c278","Type":"ContainerStarted","Data":"c71693c7e4ebebdc7ba037a19456c749d836c73028d6e89d3a1d0fe190bc1451"} Jul 15 12:07:39.291918 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:39.291860 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-zgfdl" podStartSLOduration=1.256033205 podStartE2EDuration="26.291841754s" totalImagesPullingTime="25.035808549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:07:39.289947134 +0000 UTC m=+526.703475481" watchObservedRunningTime="2026-07-15 12:07:39.291841754 +0000 UTC m=+526.705370124" Jul 15 12:07:53.156352 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:53.156266 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:53.768191 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:53.768149 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:07:59.613213 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:07:59.613171 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:07.068203 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.068161 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:07.073693 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.073667 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rfkfs"] Jul 15 12:08:07.143998 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.143953 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rfkfs"] Jul 15 12:08:07.144174 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.144088 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" Jul 15 12:08:07.146031 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.146007 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 12:08:07.176945 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.176915 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rfkfs"] Jul 15 12:08:07.179359 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.179335 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bbcee418-d400-49d3-aeed-f3f30342d28e-config-file\") pod \"limitador-limitador-64c94fd64f-rfkfs\" (UID: \"bbcee418-d400-49d3-aeed-f3f30342d28e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" Jul 15 12:08:07.179488 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.179413 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbnrs\" (UniqueName: \"kubernetes.io/projected/bbcee418-d400-49d3-aeed-f3f30342d28e-kube-api-access-nbnrs\") pod \"limitador-limitador-64c94fd64f-rfkfs\" (UID: \"bbcee418-d400-49d3-aeed-f3f30342d28e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" Jul 15 12:08:07.184370 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.184336 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:07.246490 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.246453 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:07.280702 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.280651 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bbcee418-d400-49d3-aeed-f3f30342d28e-config-file\") pod \"limitador-limitador-64c94fd64f-rfkfs\" (UID: \"bbcee418-d400-49d3-aeed-f3f30342d28e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" Jul 15 12:08:07.280702 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.280701 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbnrs\" (UniqueName: \"kubernetes.io/projected/bbcee418-d400-49d3-aeed-f3f30342d28e-kube-api-access-nbnrs\") pod \"limitador-limitador-64c94fd64f-rfkfs\" (UID: \"bbcee418-d400-49d3-aeed-f3f30342d28e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" Jul 15 12:08:07.281391 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.281368 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bbcee418-d400-49d3-aeed-f3f30342d28e-config-file\") pod \"limitador-limitador-64c94fd64f-rfkfs\" (UID: \"bbcee418-d400-49d3-aeed-f3f30342d28e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" Jul 15 12:08:07.288835 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.288802 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbnrs\" (UniqueName: \"kubernetes.io/projected/bbcee418-d400-49d3-aeed-f3f30342d28e-kube-api-access-nbnrs\") pod \"limitador-limitador-64c94fd64f-rfkfs\" (UID: \"bbcee418-d400-49d3-aeed-f3f30342d28e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" Jul 15 12:08:07.458639 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.458601 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" Jul 15 12:08:07.664263 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.664079 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rfkfs"] Jul 15 12:08:07.666982 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:08:07.666951 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcee418_d400_49d3_aeed_f3f30342d28e.slice/crio-6671facd30ce5cc93640f04c7d04ca31142421892cc4f36fddf2ca43d1355c7c WatchSource:0}: Error finding container 6671facd30ce5cc93640f04c7d04ca31142421892cc4f36fddf2ca43d1355c7c: Status 404 returned error can't find the container with id 6671facd30ce5cc93640f04c7d04ca31142421892cc4f36fddf2ca43d1355c7c Jul 15 12:08:07.874837 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.874802 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-z4pz8"] Jul 15 12:08:07.909313 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.909283 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:07.962493 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.962460 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-z4pz8"] Jul 15 12:08:07.962649 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.962555 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-z4pz8" Jul 15 12:08:07.964686 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:07.964666 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-878tq\"" Jul 15 12:08:08.026558 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:08.026523 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:08.087337 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:08.087295 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zkqs\" (UniqueName: \"kubernetes.io/projected/8c3e5a10-98e6-4f7c-8bae-a3c2800972bb-kube-api-access-4zkqs\") pod \"authorino-58cd955675-z4pz8\" (UID: \"8c3e5a10-98e6-4f7c-8bae-a3c2800972bb\") " pod="kuadrant-system/authorino-58cd955675-z4pz8" Jul 15 12:08:08.188515 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:08.188427 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zkqs\" (UniqueName: \"kubernetes.io/projected/8c3e5a10-98e6-4f7c-8bae-a3c2800972bb-kube-api-access-4zkqs\") pod \"authorino-58cd955675-z4pz8\" (UID: \"8c3e5a10-98e6-4f7c-8bae-a3c2800972bb\") " pod="kuadrant-system/authorino-58cd955675-z4pz8" Jul 15 12:08:08.195885 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:08.195862 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zkqs\" (UniqueName: \"kubernetes.io/projected/8c3e5a10-98e6-4f7c-8bae-a3c2800972bb-kube-api-access-4zkqs\") pod \"authorino-58cd955675-z4pz8\" (UID: \"8c3e5a10-98e6-4f7c-8bae-a3c2800972bb\") " pod="kuadrant-system/authorino-58cd955675-z4pz8" Jul 15 12:08:08.272460 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:08.272427 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-z4pz8" Jul 15 12:08:08.394117 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:08.394082 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" event={"ID":"bbcee418-d400-49d3-aeed-f3f30342d28e","Type":"ContainerStarted","Data":"6671facd30ce5cc93640f04c7d04ca31142421892cc4f36fddf2ca43d1355c7c"} Jul 15 12:08:08.404372 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:08.404351 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-z4pz8"] Jul 15 12:08:08.406556 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:08:08.406532 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c3e5a10_98e6_4f7c_8bae_a3c2800972bb.slice/crio-e03d90f543adefafeb443f5c305a3afcc276eec9f2622e920312aeba3f0cd9ca WatchSource:0}: Error finding container e03d90f543adefafeb443f5c305a3afcc276eec9f2622e920312aeba3f0cd9ca: Status 404 returned error can't find the container with id e03d90f543adefafeb443f5c305a3afcc276eec9f2622e920312aeba3f0cd9ca Jul 15 12:08:09.403432 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:09.403327 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-z4pz8" event={"ID":"8c3e5a10-98e6-4f7c-8bae-a3c2800972bb","Type":"ContainerStarted","Data":"e03d90f543adefafeb443f5c305a3afcc276eec9f2622e920312aeba3f0cd9ca"} Jul 15 12:08:12.166249 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:12.166216 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-z4pz8"] Jul 15 12:08:13.423795 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:13.423758 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-z4pz8" event={"ID":"8c3e5a10-98e6-4f7c-8bae-a3c2800972bb","Type":"ContainerStarted","Data":"12473424f9e9d68fad8d73e2ba42f9d41f7e9647ade9e9fedd2f3b72e0270cb2"} Jul 15 12:08:13.423795 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:13.423775 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-z4pz8" podUID="8c3e5a10-98e6-4f7c-8bae-a3c2800972bb" containerName="authorino" containerID="cri-o://12473424f9e9d68fad8d73e2ba42f9d41f7e9647ade9e9fedd2f3b72e0270cb2" gracePeriod=30 Jul 15 12:08:13.425462 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:13.425433 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" event={"ID":"bbcee418-d400-49d3-aeed-f3f30342d28e","Type":"ContainerStarted","Data":"9587f9a7a183fb87cc8347538ffbd087e70ea392fed0689ad883d1d8d01e1f1d"} Jul 15 12:08:13.425595 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:13.425499 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" Jul 15 12:08:13.439801 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:13.439733 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-z4pz8" podStartSLOduration=2.36419696 podStartE2EDuration="6.439717049s" totalImagesPullingTime="4.075520089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:13.438705905 +0000 UTC m=+560.852234258" watchObservedRunningTime="2026-07-15 12:08:13.439717049 +0000 UTC m=+560.853245399" Jul 15 12:08:13.454731 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:13.454681 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" podStartSLOduration=1.64103499 podStartE2EDuration="6.45466371s" totalImagesPullingTime="4.81362872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:13.453220156 +0000 UTC m=+560.866748506" watchObservedRunningTime="2026-07-15 12:08:13.45466371 +0000 UTC m=+560.868192062" Jul 15 12:08:13.667962 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:13.667932 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-z4pz8" Jul 15 12:08:13.743132 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:13.743044 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zkqs\" (UniqueName: \"kubernetes.io/projected/8c3e5a10-98e6-4f7c-8bae-a3c2800972bb-kube-api-access-4zkqs\") pod \"8c3e5a10-98e6-4f7c-8bae-a3c2800972bb\" (UID: \"8c3e5a10-98e6-4f7c-8bae-a3c2800972bb\") " Jul 15 12:08:13.745342 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:13.745305 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3e5a10-98e6-4f7c-8bae-a3c2800972bb-kube-api-access-4zkqs" (OuterVolumeSpecName: "kube-api-access-4zkqs") pod "8c3e5a10-98e6-4f7c-8bae-a3c2800972bb" (UID: "8c3e5a10-98e6-4f7c-8bae-a3c2800972bb"). InnerVolumeSpecName "kube-api-access-4zkqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:08:13.844714 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:13.844674 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zkqs\" (UniqueName: \"kubernetes.io/projected/8c3e5a10-98e6-4f7c-8bae-a3c2800972bb-kube-api-access-4zkqs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:08:14.431330 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:14.431300 2570 generic.go:356] "Generic (PLEG): container finished" podID="8c3e5a10-98e6-4f7c-8bae-a3c2800972bb" containerID="12473424f9e9d68fad8d73e2ba42f9d41f7e9647ade9e9fedd2f3b72e0270cb2" exitCode=0 Jul 15 12:08:14.431792 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:14.431339 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-z4pz8" Jul 15 12:08:14.431792 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:14.431367 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-z4pz8" event={"ID":"8c3e5a10-98e6-4f7c-8bae-a3c2800972bb","Type":"ContainerDied","Data":"12473424f9e9d68fad8d73e2ba42f9d41f7e9647ade9e9fedd2f3b72e0270cb2"} Jul 15 12:08:14.431792 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:14.431397 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-z4pz8" event={"ID":"8c3e5a10-98e6-4f7c-8bae-a3c2800972bb","Type":"ContainerDied","Data":"e03d90f543adefafeb443f5c305a3afcc276eec9f2622e920312aeba3f0cd9ca"} Jul 15 12:08:14.431792 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:14.431413 2570 scope.go:117] "RemoveContainer" containerID="12473424f9e9d68fad8d73e2ba42f9d41f7e9647ade9e9fedd2f3b72e0270cb2" Jul 15 12:08:14.440666 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:14.440645 2570 scope.go:117] "RemoveContainer" containerID="12473424f9e9d68fad8d73e2ba42f9d41f7e9647ade9e9fedd2f3b72e0270cb2" Jul 15 12:08:14.440966 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:08:14.440944 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12473424f9e9d68fad8d73e2ba42f9d41f7e9647ade9e9fedd2f3b72e0270cb2\": container with ID starting with 12473424f9e9d68fad8d73e2ba42f9d41f7e9647ade9e9fedd2f3b72e0270cb2 not found: ID does not exist" containerID="12473424f9e9d68fad8d73e2ba42f9d41f7e9647ade9e9fedd2f3b72e0270cb2" Jul 15 12:08:14.441046 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:14.440976 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12473424f9e9d68fad8d73e2ba42f9d41f7e9647ade9e9fedd2f3b72e0270cb2"} err="failed to get container status \"12473424f9e9d68fad8d73e2ba42f9d41f7e9647ade9e9fedd2f3b72e0270cb2\": rpc error: code = NotFound desc = could not find container \"12473424f9e9d68fad8d73e2ba42f9d41f7e9647ade9e9fedd2f3b72e0270cb2\": container with ID starting with 12473424f9e9d68fad8d73e2ba42f9d41f7e9647ade9e9fedd2f3b72e0270cb2 not found: ID does not exist" Jul 15 12:08:14.451281 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:14.451256 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-z4pz8"] Jul 15 12:08:14.452816 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:14.452793 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-z4pz8"] Jul 15 12:08:15.091612 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:15.091575 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c3e5a10-98e6-4f7c-8bae-a3c2800972bb" path="/var/lib/kubelet/pods/8c3e5a10-98e6-4f7c-8bae-a3c2800972bb/volumes" Jul 15 12:08:22.003042 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:21.999515 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rfkfs"] Jul 15 12:08:22.003042 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:21.999850 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" podUID="bbcee418-d400-49d3-aeed-f3f30342d28e" containerName="limitador" containerID="cri-o://9587f9a7a183fb87cc8347538ffbd087e70ea392fed0689ad883d1d8d01e1f1d" gracePeriod=30 Jul 15 12:08:22.004143 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:22.003923 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" Jul 15 12:08:22.468301 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:22.468270 2570 generic.go:356] "Generic (PLEG): container finished" podID="bbcee418-d400-49d3-aeed-f3f30342d28e" containerID="9587f9a7a183fb87cc8347538ffbd087e70ea392fed0689ad883d1d8d01e1f1d" exitCode=0 Jul 15 12:08:22.468457 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:22.468345 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" event={"ID":"bbcee418-d400-49d3-aeed-f3f30342d28e","Type":"ContainerDied","Data":"9587f9a7a183fb87cc8347538ffbd087e70ea392fed0689ad883d1d8d01e1f1d"} Jul 15 12:08:22.946855 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:22.946829 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" Jul 15 12:08:23.124244 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:23.124221 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bbcee418-d400-49d3-aeed-f3f30342d28e-config-file\") pod \"bbcee418-d400-49d3-aeed-f3f30342d28e\" (UID: \"bbcee418-d400-49d3-aeed-f3f30342d28e\") " Jul 15 12:08:23.124599 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:23.124291 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbnrs\" (UniqueName: \"kubernetes.io/projected/bbcee418-d400-49d3-aeed-f3f30342d28e-kube-api-access-nbnrs\") pod \"bbcee418-d400-49d3-aeed-f3f30342d28e\" (UID: \"bbcee418-d400-49d3-aeed-f3f30342d28e\") " Jul 15 12:08:23.124599 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:23.124550 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbcee418-d400-49d3-aeed-f3f30342d28e-config-file" (OuterVolumeSpecName: "config-file") pod "bbcee418-d400-49d3-aeed-f3f30342d28e" (UID: "bbcee418-d400-49d3-aeed-f3f30342d28e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:08:23.126395 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:23.126378 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbcee418-d400-49d3-aeed-f3f30342d28e-kube-api-access-nbnrs" (OuterVolumeSpecName: "kube-api-access-nbnrs") pod "bbcee418-d400-49d3-aeed-f3f30342d28e" (UID: "bbcee418-d400-49d3-aeed-f3f30342d28e"). InnerVolumeSpecName "kube-api-access-nbnrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:08:23.225493 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:23.225465 2570 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bbcee418-d400-49d3-aeed-f3f30342d28e-config-file\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:08:23.225493 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:23.225489 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbnrs\" (UniqueName: \"kubernetes.io/projected/bbcee418-d400-49d3-aeed-f3f30342d28e-kube-api-access-nbnrs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:08:23.474058 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:23.473979 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" Jul 15 12:08:23.474214 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:23.473979 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-rfkfs" event={"ID":"bbcee418-d400-49d3-aeed-f3f30342d28e","Type":"ContainerDied","Data":"6671facd30ce5cc93640f04c7d04ca31142421892cc4f36fddf2ca43d1355c7c"} Jul 15 12:08:23.474214 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:23.474123 2570 scope.go:117] "RemoveContainer" containerID="9587f9a7a183fb87cc8347538ffbd087e70ea392fed0689ad883d1d8d01e1f1d" Jul 15 12:08:23.496034 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:23.496008 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rfkfs"] Jul 15 12:08:23.498290 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:23.498268 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-rfkfs"] Jul 15 12:08:25.091810 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:25.091778 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbcee418-d400-49d3-aeed-f3f30342d28e" path="/var/lib/kubelet/pods/bbcee418-d400-49d3-aeed-f3f30342d28e/volumes" Jul 15 12:08:26.583851 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:26.583820 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:26.681947 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:26.681894 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:27.037299 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.037270 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj"] Jul 15 12:08:27.037628 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.037616 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbcee418-d400-49d3-aeed-f3f30342d28e" containerName="limitador" Jul 15 12:08:27.037672 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.037630 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbcee418-d400-49d3-aeed-f3f30342d28e" containerName="limitador" Jul 15 12:08:27.037672 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.037652 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c3e5a10-98e6-4f7c-8bae-a3c2800972bb" containerName="authorino" Jul 15 12:08:27.037672 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.037659 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c3e5a10-98e6-4f7c-8bae-a3c2800972bb" containerName="authorino" Jul 15 12:08:27.037776 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.037710 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbcee418-d400-49d3-aeed-f3f30342d28e" containerName="limitador" Jul 15 12:08:27.037776 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.037720 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c3e5a10-98e6-4f7c-8bae-a3c2800972bb" containerName="authorino" Jul 15 12:08:27.041629 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.041609 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj" Jul 15 12:08:27.043513 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.043487 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 12:08:27.043620 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.043582 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 15 12:08:27.043620 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.043582 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 12:08:27.043944 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.043911 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-5qb5l\"" Jul 15 12:08:27.049813 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.049792 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj"] Jul 15 12:08:27.068859 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.068831 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:27.162335 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.162301 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6lv5\" (UniqueName: \"kubernetes.io/projected/f572fcfc-1218-42bf-bc41-68056a69f4b5-kube-api-access-p6lv5\") pod \"postgres-69fb4594cf-5q2pj\" (UID: \"f572fcfc-1218-42bf-bc41-68056a69f4b5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj" Jul 15 12:08:27.162503 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.162396 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f572fcfc-1218-42bf-bc41-68056a69f4b5-data\") pod \"postgres-69fb4594cf-5q2pj\" (UID: \"f572fcfc-1218-42bf-bc41-68056a69f4b5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj" Jul 15 12:08:27.263514 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.263482 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f572fcfc-1218-42bf-bc41-68056a69f4b5-data\") pod \"postgres-69fb4594cf-5q2pj\" (UID: \"f572fcfc-1218-42bf-bc41-68056a69f4b5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj" Jul 15 12:08:27.263675 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.263529 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6lv5\" (UniqueName: \"kubernetes.io/projected/f572fcfc-1218-42bf-bc41-68056a69f4b5-kube-api-access-p6lv5\") pod \"postgres-69fb4594cf-5q2pj\" (UID: \"f572fcfc-1218-42bf-bc41-68056a69f4b5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj" Jul 15 12:08:27.263894 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.263873 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f572fcfc-1218-42bf-bc41-68056a69f4b5-data\") pod \"postgres-69fb4594cf-5q2pj\" (UID: \"f572fcfc-1218-42bf-bc41-68056a69f4b5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj" Jul 15 12:08:27.289705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.289635 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6lv5\" (UniqueName: \"kubernetes.io/projected/f572fcfc-1218-42bf-bc41-68056a69f4b5-kube-api-access-p6lv5\") pod \"postgres-69fb4594cf-5q2pj\" (UID: \"f572fcfc-1218-42bf-bc41-68056a69f4b5\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj" Jul 15 12:08:27.354091 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.354057 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj" Jul 15 12:08:27.483283 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.483251 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj"] Jul 15 12:08:27.486235 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:08:27.486205 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf572fcfc_1218_42bf_bc41_68056a69f4b5.slice/crio-4c0388dcd8217d93b830da554501b82a63bd3d6cafc1d202b8e2ee89c2e0421f WatchSource:0}: Error finding container 4c0388dcd8217d93b830da554501b82a63bd3d6cafc1d202b8e2ee89c2e0421f: Status 404 returned error can't find the container with id 4c0388dcd8217d93b830da554501b82a63bd3d6cafc1d202b8e2ee89c2e0421f Jul 15 12:08:27.493692 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.493661 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj" event={"ID":"f572fcfc-1218-42bf-bc41-68056a69f4b5","Type":"ContainerStarted","Data":"4c0388dcd8217d93b830da554501b82a63bd3d6cafc1d202b8e2ee89c2e0421f"} Jul 15 12:08:27.715867 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:27.715830 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:33.530669 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:33.530635 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj" event={"ID":"f572fcfc-1218-42bf-bc41-68056a69f4b5","Type":"ContainerStarted","Data":"2f8570fb39cf7377aaa1e1290810222cb236bff6d6139cc0f56524c576682694"} Jul 15 12:08:33.531060 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:33.530767 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj" Jul 15 12:08:33.548426 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:33.548375 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj" podStartSLOduration=1.2860822650000001 podStartE2EDuration="6.548357744s" totalImagesPullingTime="5.262275479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:33.545966368 +0000 UTC m=+580.959494732" watchObservedRunningTime="2026-07-15 12:08:33.548357744 +0000 UTC m=+580.961886094" Jul 15 12:08:39.565142 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:39.565115 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5q2pj" Jul 15 12:08:39.931727 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:39.931682 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:40.131670 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.131636 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bpk8d"] Jul 15 12:08:40.137587 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.137564 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bpk8d" Jul 15 12:08:40.139338 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.139320 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-878tq\"" Jul 15 12:08:40.141826 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.141799 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bpk8d"] Jul 15 12:08:40.170702 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.170656 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dv2\" (UniqueName: \"kubernetes.io/projected/73c3a52c-75de-4622-aa16-99665859c102-kube-api-access-24dv2\") pod \"authorino-5d4b67c446-bpk8d\" (UID: \"73c3a52c-75de-4622-aa16-99665859c102\") " pod="kuadrant-system/authorino-5d4b67c446-bpk8d" Jul 15 12:08:40.271555 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.271474 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dv2\" (UniqueName: \"kubernetes.io/projected/73c3a52c-75de-4622-aa16-99665859c102-kube-api-access-24dv2\") pod \"authorino-5d4b67c446-bpk8d\" (UID: \"73c3a52c-75de-4622-aa16-99665859c102\") " pod="kuadrant-system/authorino-5d4b67c446-bpk8d" Jul 15 12:08:40.280992 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.280959 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dv2\" (UniqueName: \"kubernetes.io/projected/73c3a52c-75de-4622-aa16-99665859c102-kube-api-access-24dv2\") pod \"authorino-5d4b67c446-bpk8d\" (UID: \"73c3a52c-75de-4622-aa16-99665859c102\") " pod="kuadrant-system/authorino-5d4b67c446-bpk8d" Jul 15 12:08:40.374917 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.374881 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bpk8d"] Jul 15 12:08:40.375116 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.375105 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bpk8d" Jul 15 12:08:40.400298 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.400270 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8d7f978d5-mkgmd"] Jul 15 12:08:40.405056 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.405033 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8d7f978d5-mkgmd" Jul 15 12:08:40.422381 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.422242 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8d7f978d5-mkgmd"] Jul 15 12:08:40.473460 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.473409 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g5sc\" (UniqueName: \"kubernetes.io/projected/99022f18-7d49-4928-968e-cecaa5c80381-kube-api-access-6g5sc\") pod \"authorino-8d7f978d5-mkgmd\" (UID: \"99022f18-7d49-4928-968e-cecaa5c80381\") " pod="kuadrant-system/authorino-8d7f978d5-mkgmd" Jul 15 12:08:40.487774 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.487715 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:40.513871 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.513845 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bpk8d"] Jul 15 12:08:40.516507 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:08:40.516480 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73c3a52c_75de_4622_aa16_99665859c102.slice/crio-7c85e650956b117667d0e550d3daa65980ebedf4a0ebefa7477e8387c94d15c2 WatchSource:0}: Error finding container 7c85e650956b117667d0e550d3daa65980ebedf4a0ebefa7477e8387c94d15c2: Status 404 returned error can't find the container with id 7c85e650956b117667d0e550d3daa65980ebedf4a0ebefa7477e8387c94d15c2 Jul 15 12:08:40.562094 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.562022 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bpk8d" event={"ID":"73c3a52c-75de-4622-aa16-99665859c102","Type":"ContainerStarted","Data":"7c85e650956b117667d0e550d3daa65980ebedf4a0ebefa7477e8387c94d15c2"} Jul 15 12:08:40.573971 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.573949 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g5sc\" (UniqueName: \"kubernetes.io/projected/99022f18-7d49-4928-968e-cecaa5c80381-kube-api-access-6g5sc\") pod \"authorino-8d7f978d5-mkgmd\" (UID: \"99022f18-7d49-4928-968e-cecaa5c80381\") " pod="kuadrant-system/authorino-8d7f978d5-mkgmd" Jul 15 12:08:40.581278 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.581256 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g5sc\" (UniqueName: \"kubernetes.io/projected/99022f18-7d49-4928-968e-cecaa5c80381-kube-api-access-6g5sc\") pod \"authorino-8d7f978d5-mkgmd\" (UID: \"99022f18-7d49-4928-968e-cecaa5c80381\") " pod="kuadrant-system/authorino-8d7f978d5-mkgmd" Jul 15 12:08:40.635314 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.635274 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8d7f978d5-mkgmd"] Jul 15 12:08:40.635556 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.635543 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8d7f978d5-mkgmd" Jul 15 12:08:40.768464 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:40.768437 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8d7f978d5-mkgmd"] Jul 15 12:08:40.770513 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:08:40.770485 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99022f18_7d49_4928_968e_cecaa5c80381.slice/crio-61f86ad28eda04c142e941624293766d77f6b12044d4a885f4085648030aa5aa WatchSource:0}: Error finding container 61f86ad28eda04c142e941624293766d77f6b12044d4a885f4085648030aa5aa: Status 404 returned error can't find the container with id 61f86ad28eda04c142e941624293766d77f6b12044d4a885f4085648030aa5aa Jul 15 12:08:41.568863 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:41.568732 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8d7f978d5-mkgmd" event={"ID":"99022f18-7d49-4928-968e-cecaa5c80381","Type":"ContainerStarted","Data":"66ebc56c61a48f7041bed25feb3758bfb0d4345a065fbd7388ce875860e397c4"} Jul 15 12:08:41.568863 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:41.568817 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8d7f978d5-mkgmd" event={"ID":"99022f18-7d49-4928-968e-cecaa5c80381","Type":"ContainerStarted","Data":"61f86ad28eda04c142e941624293766d77f6b12044d4a885f4085648030aa5aa"} Jul 15 12:08:41.568863 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:41.568803 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8d7f978d5-mkgmd" podUID="99022f18-7d49-4928-968e-cecaa5c80381" containerName="authorino" containerID="cri-o://66ebc56c61a48f7041bed25feb3758bfb0d4345a065fbd7388ce875860e397c4" gracePeriod=30 Jul 15 12:08:41.570353 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:41.570328 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bpk8d" event={"ID":"73c3a52c-75de-4622-aa16-99665859c102","Type":"ContainerStarted","Data":"8d8f6f7898499e00c0aa899417d15eae6a08f776ec89b9fde7ab19db11450ab7"} Jul 15 12:08:41.570473 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:41.570376 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-bpk8d" podUID="73c3a52c-75de-4622-aa16-99665859c102" containerName="authorino" containerID="cri-o://8d8f6f7898499e00c0aa899417d15eae6a08f776ec89b9fde7ab19db11450ab7" gracePeriod=30 Jul 15 12:08:41.586406 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:41.586356 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8d7f978d5-mkgmd" podStartSLOduration=1.234555864 podStartE2EDuration="1.586335324s" totalImagesPullingTime="351.77946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:41.583330066 +0000 UTC m=+588.996858417" watchObservedRunningTime="2026-07-15 12:08:41.586335324 +0000 UTC m=+588.999863676" Jul 15 12:08:41.602660 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:41.602611 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-bpk8d" podStartSLOduration=1.2582103199999999 podStartE2EDuration="1.602592138s" totalImagesPullingTime="344.381818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:41.601395997 +0000 UTC m=+589.014924350" watchObservedRunningTime="2026-07-15 12:08:41.602592138 +0000 UTC m=+589.016120541" Jul 15 12:08:41.836368 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:41.836343 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bpk8d" Jul 15 12:08:41.839152 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:41.839133 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8d7f978d5-mkgmd" Jul 15 12:08:41.885922 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:41.885883 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24dv2\" (UniqueName: \"kubernetes.io/projected/73c3a52c-75de-4622-aa16-99665859c102-kube-api-access-24dv2\") pod \"73c3a52c-75de-4622-aa16-99665859c102\" (UID: \"73c3a52c-75de-4622-aa16-99665859c102\") " Jul 15 12:08:41.885922 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:41.885920 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g5sc\" (UniqueName: \"kubernetes.io/projected/99022f18-7d49-4928-968e-cecaa5c80381-kube-api-access-6g5sc\") pod \"99022f18-7d49-4928-968e-cecaa5c80381\" (UID: \"99022f18-7d49-4928-968e-cecaa5c80381\") " Jul 15 12:08:41.888163 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:41.888137 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c3a52c-75de-4622-aa16-99665859c102-kube-api-access-24dv2" (OuterVolumeSpecName: "kube-api-access-24dv2") pod "73c3a52c-75de-4622-aa16-99665859c102" (UID: "73c3a52c-75de-4622-aa16-99665859c102"). InnerVolumeSpecName "kube-api-access-24dv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:08:41.888256 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:41.888171 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99022f18-7d49-4928-968e-cecaa5c80381-kube-api-access-6g5sc" (OuterVolumeSpecName: "kube-api-access-6g5sc") pod "99022f18-7d49-4928-968e-cecaa5c80381" (UID: "99022f18-7d49-4928-968e-cecaa5c80381"). InnerVolumeSpecName "kube-api-access-6g5sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:08:41.987225 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:41.987179 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24dv2\" (UniqueName: \"kubernetes.io/projected/73c3a52c-75de-4622-aa16-99665859c102-kube-api-access-24dv2\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:08:41.987225 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:41.987213 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g5sc\" (UniqueName: \"kubernetes.io/projected/99022f18-7d49-4928-968e-cecaa5c80381-kube-api-access-6g5sc\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:08:42.575678 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.575584 2570 generic.go:356] "Generic (PLEG): container finished" podID="73c3a52c-75de-4622-aa16-99665859c102" containerID="8d8f6f7898499e00c0aa899417d15eae6a08f776ec89b9fde7ab19db11450ab7" exitCode=0 Jul 15 12:08:42.575678 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.575629 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bpk8d" Jul 15 12:08:42.575910 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.575675 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bpk8d" event={"ID":"73c3a52c-75de-4622-aa16-99665859c102","Type":"ContainerDied","Data":"8d8f6f7898499e00c0aa899417d15eae6a08f776ec89b9fde7ab19db11450ab7"} Jul 15 12:08:42.575910 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.575716 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bpk8d" event={"ID":"73c3a52c-75de-4622-aa16-99665859c102","Type":"ContainerDied","Data":"7c85e650956b117667d0e550d3daa65980ebedf4a0ebefa7477e8387c94d15c2"} Jul 15 12:08:42.575910 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.575732 2570 scope.go:117] "RemoveContainer" containerID="8d8f6f7898499e00c0aa899417d15eae6a08f776ec89b9fde7ab19db11450ab7" Jul 15 12:08:42.577050 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.576975 2570 generic.go:356] "Generic (PLEG): container finished" podID="99022f18-7d49-4928-968e-cecaa5c80381" containerID="66ebc56c61a48f7041bed25feb3758bfb0d4345a065fbd7388ce875860e397c4" exitCode=0 Jul 15 12:08:42.577050 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.577011 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8d7f978d5-mkgmd" event={"ID":"99022f18-7d49-4928-968e-cecaa5c80381","Type":"ContainerDied","Data":"66ebc56c61a48f7041bed25feb3758bfb0d4345a065fbd7388ce875860e397c4"} Jul 15 12:08:42.577050 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.577030 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8d7f978d5-mkgmd" Jul 15 12:08:42.577050 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.577044 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8d7f978d5-mkgmd" event={"ID":"99022f18-7d49-4928-968e-cecaa5c80381","Type":"ContainerDied","Data":"61f86ad28eda04c142e941624293766d77f6b12044d4a885f4085648030aa5aa"} Jul 15 12:08:42.584822 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.584808 2570 scope.go:117] "RemoveContainer" containerID="8d8f6f7898499e00c0aa899417d15eae6a08f776ec89b9fde7ab19db11450ab7" Jul 15 12:08:42.585199 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:08:42.585068 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d8f6f7898499e00c0aa899417d15eae6a08f776ec89b9fde7ab19db11450ab7\": container with ID starting with 8d8f6f7898499e00c0aa899417d15eae6a08f776ec89b9fde7ab19db11450ab7 not found: ID does not exist" containerID="8d8f6f7898499e00c0aa899417d15eae6a08f776ec89b9fde7ab19db11450ab7" Jul 15 12:08:42.585199 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.585101 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8f6f7898499e00c0aa899417d15eae6a08f776ec89b9fde7ab19db11450ab7"} err="failed to get container status \"8d8f6f7898499e00c0aa899417d15eae6a08f776ec89b9fde7ab19db11450ab7\": rpc error: code = NotFound desc = could not find container \"8d8f6f7898499e00c0aa899417d15eae6a08f776ec89b9fde7ab19db11450ab7\": container with ID starting with 8d8f6f7898499e00c0aa899417d15eae6a08f776ec89b9fde7ab19db11450ab7 not found: ID does not exist" Jul 15 12:08:42.585199 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.585123 2570 scope.go:117] "RemoveContainer" containerID="66ebc56c61a48f7041bed25feb3758bfb0d4345a065fbd7388ce875860e397c4" Jul 15 12:08:42.593128 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.593110 2570 scope.go:117] "RemoveContainer" containerID="66ebc56c61a48f7041bed25feb3758bfb0d4345a065fbd7388ce875860e397c4" Jul 15 12:08:42.593398 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:08:42.593382 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ebc56c61a48f7041bed25feb3758bfb0d4345a065fbd7388ce875860e397c4\": container with ID starting with 66ebc56c61a48f7041bed25feb3758bfb0d4345a065fbd7388ce875860e397c4 not found: ID does not exist" containerID="66ebc56c61a48f7041bed25feb3758bfb0d4345a065fbd7388ce875860e397c4" Jul 15 12:08:42.593434 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.593405 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ebc56c61a48f7041bed25feb3758bfb0d4345a065fbd7388ce875860e397c4"} err="failed to get container status \"66ebc56c61a48f7041bed25feb3758bfb0d4345a065fbd7388ce875860e397c4\": rpc error: code = NotFound desc = could not find container \"66ebc56c61a48f7041bed25feb3758bfb0d4345a065fbd7388ce875860e397c4\": container with ID starting with 66ebc56c61a48f7041bed25feb3758bfb0d4345a065fbd7388ce875860e397c4 not found: ID does not exist" Jul 15 12:08:42.600565 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.600541 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bpk8d"] Jul 15 12:08:42.603886 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.603864 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bpk8d"] Jul 15 12:08:42.612320 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.612299 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8d7f978d5-mkgmd"] Jul 15 12:08:42.618260 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:42.618233 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8d7f978d5-mkgmd"] Jul 15 12:08:43.091868 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:43.091839 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73c3a52c-75de-4622-aa16-99665859c102" path="/var/lib/kubelet/pods/73c3a52c-75de-4622-aa16-99665859c102/volumes" Jul 15 12:08:43.092164 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:43.092152 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99022f18-7d49-4928-968e-cecaa5c80381" path="/var/lib/kubelet/pods/99022f18-7d49-4928-968e-cecaa5c80381/volumes" Jul 15 12:08:47.071049 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.071007 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-76897f7847-gzs6x"] Jul 15 12:08:47.071500 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.071340 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c3a52c-75de-4622-aa16-99665859c102" containerName="authorino" Jul 15 12:08:47.071500 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.071351 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c3a52c-75de-4622-aa16-99665859c102" containerName="authorino" Jul 15 12:08:47.071500 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.071362 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99022f18-7d49-4928-968e-cecaa5c80381" containerName="authorino" Jul 15 12:08:47.071500 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.071367 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="99022f18-7d49-4928-968e-cecaa5c80381" containerName="authorino" Jul 15 12:08:47.071500 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.071428 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c3a52c-75de-4622-aa16-99665859c102" containerName="authorino" Jul 15 12:08:47.071500 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.071435 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="99022f18-7d49-4928-968e-cecaa5c80381" containerName="authorino" Jul 15 12:08:47.085991 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.085959 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:47.181530 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.181497 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:47.183165 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.183145 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76897f7847-gzs6x" Jul 15 12:08:47.185616 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.185589 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 15 12:08:47.185772 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.185590 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 15 12:08:47.185772 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.185633 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wsx7r\"" Jul 15 12:08:47.188986 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.188963 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76897f7847-gzs6x"] Jul 15 12:08:47.232043 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.232003 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72198a09-0f56-4462-9258-e11ec9bdec0f-webhook-cert\") pod \"maas-controller-76897f7847-gzs6x\" (UID: \"72198a09-0f56-4462-9258-e11ec9bdec0f\") " pod="opendatahub/maas-controller-76897f7847-gzs6x" Jul 15 12:08:47.232262 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.232086 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9qqw\" (UniqueName: \"kubernetes.io/projected/72198a09-0f56-4462-9258-e11ec9bdec0f-kube-api-access-d9qqw\") pod \"maas-controller-76897f7847-gzs6x\" (UID: \"72198a09-0f56-4462-9258-e11ec9bdec0f\") " pod="opendatahub/maas-controller-76897f7847-gzs6x" Jul 15 12:08:47.256787 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.256726 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55594846d9-5mxfn"] Jul 15 12:08:47.280366 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.280340 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55594846d9-5mxfn" Jul 15 12:08:47.289587 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.289556 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55594846d9-5mxfn"] Jul 15 12:08:47.320076 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.320040 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:47.333297 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.333223 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2lh6\" (UniqueName: \"kubernetes.io/projected/a5d9e822-bb8b-44e5-889a-322898e09eea-kube-api-access-x2lh6\") pod \"maas-controller-55594846d9-5mxfn\" (UID: \"a5d9e822-bb8b-44e5-889a-322898e09eea\") " pod="opendatahub/maas-controller-55594846d9-5mxfn" Jul 15 12:08:47.333297 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.333276 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72198a09-0f56-4462-9258-e11ec9bdec0f-webhook-cert\") pod \"maas-controller-76897f7847-gzs6x\" (UID: \"72198a09-0f56-4462-9258-e11ec9bdec0f\") " pod="opendatahub/maas-controller-76897f7847-gzs6x" Jul 15 12:08:47.333470 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.333304 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9qqw\" (UniqueName: \"kubernetes.io/projected/72198a09-0f56-4462-9258-e11ec9bdec0f-kube-api-access-d9qqw\") pod \"maas-controller-76897f7847-gzs6x\" (UID: \"72198a09-0f56-4462-9258-e11ec9bdec0f\") " pod="opendatahub/maas-controller-76897f7847-gzs6x" Jul 15 12:08:47.333470 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.333359 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d9e822-bb8b-44e5-889a-322898e09eea-webhook-cert\") pod \"maas-controller-55594846d9-5mxfn\" (UID: \"a5d9e822-bb8b-44e5-889a-322898e09eea\") " pod="opendatahub/maas-controller-55594846d9-5mxfn" Jul 15 12:08:47.335791 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.335770 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72198a09-0f56-4462-9258-e11ec9bdec0f-webhook-cert\") pod \"maas-controller-76897f7847-gzs6x\" (UID: \"72198a09-0f56-4462-9258-e11ec9bdec0f\") " pod="opendatahub/maas-controller-76897f7847-gzs6x" Jul 15 12:08:47.341075 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.341051 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9qqw\" (UniqueName: \"kubernetes.io/projected/72198a09-0f56-4462-9258-e11ec9bdec0f-kube-api-access-d9qqw\") pod \"maas-controller-76897f7847-gzs6x\" (UID: \"72198a09-0f56-4462-9258-e11ec9bdec0f\") " pod="opendatahub/maas-controller-76897f7847-gzs6x" Jul 15 12:08:47.434119 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.434081 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2lh6\" (UniqueName: \"kubernetes.io/projected/a5d9e822-bb8b-44e5-889a-322898e09eea-kube-api-access-x2lh6\") pod \"maas-controller-55594846d9-5mxfn\" (UID: \"a5d9e822-bb8b-44e5-889a-322898e09eea\") " pod="opendatahub/maas-controller-55594846d9-5mxfn" Jul 15 12:08:47.434289 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.434156 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d9e822-bb8b-44e5-889a-322898e09eea-webhook-cert\") pod \"maas-controller-55594846d9-5mxfn\" (UID: \"a5d9e822-bb8b-44e5-889a-322898e09eea\") " pod="opendatahub/maas-controller-55594846d9-5mxfn" Jul 15 12:08:47.436847 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.436822 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d9e822-bb8b-44e5-889a-322898e09eea-webhook-cert\") pod \"maas-controller-55594846d9-5mxfn\" (UID: \"a5d9e822-bb8b-44e5-889a-322898e09eea\") " pod="opendatahub/maas-controller-55594846d9-5mxfn" Jul 15 12:08:47.442079 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.442051 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2lh6\" (UniqueName: \"kubernetes.io/projected/a5d9e822-bb8b-44e5-889a-322898e09eea-kube-api-access-x2lh6\") pod \"maas-controller-55594846d9-5mxfn\" (UID: \"a5d9e822-bb8b-44e5-889a-322898e09eea\") " pod="opendatahub/maas-controller-55594846d9-5mxfn" Jul 15 12:08:47.493990 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.493951 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76897f7847-gzs6x" Jul 15 12:08:47.591866 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.591408 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55594846d9-5mxfn" Jul 15 12:08:47.643450 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.643426 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-76897f7847-gzs6x"] Jul 15 12:08:47.646457 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:08:47.646431 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72198a09_0f56_4462_9258_e11ec9bdec0f.slice/crio-a200c3c48447030903293237df08694aa8d8fe7c78e532466b0dd21123c4f825 WatchSource:0}: Error finding container a200c3c48447030903293237df08694aa8d8fe7c78e532466b0dd21123c4f825: Status 404 returned error can't find the container with id a200c3c48447030903293237df08694aa8d8fe7c78e532466b0dd21123c4f825 Jul 15 12:08:47.746373 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:47.746341 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55594846d9-5mxfn"] Jul 15 12:08:47.749618 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:08:47.749592 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d9e822_bb8b_44e5_889a_322898e09eea.slice/crio-d42feb7a49f8b1562aaa61efec37f233235da6f7c6f7686ec50877e35cc7d752 WatchSource:0}: Error finding container d42feb7a49f8b1562aaa61efec37f233235da6f7c6f7686ec50877e35cc7d752: Status 404 returned error can't find the container with id d42feb7a49f8b1562aaa61efec37f233235da6f7c6f7686ec50877e35cc7d752 Jul 15 12:08:48.609366 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:48.609289 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55594846d9-5mxfn" event={"ID":"a5d9e822-bb8b-44e5-889a-322898e09eea","Type":"ContainerStarted","Data":"d42feb7a49f8b1562aaa61efec37f233235da6f7c6f7686ec50877e35cc7d752"} Jul 15 12:08:48.610806 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:48.610775 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76897f7847-gzs6x" event={"ID":"72198a09-0f56-4462-9258-e11ec9bdec0f","Type":"ContainerStarted","Data":"a200c3c48447030903293237df08694aa8d8fe7c78e532466b0dd21123c4f825"} Jul 15 12:08:50.444598 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:50.444558 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:50.552978 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:50.552882 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:50.624804 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:50.624766 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76897f7847-gzs6x" event={"ID":"72198a09-0f56-4462-9258-e11ec9bdec0f","Type":"ContainerStarted","Data":"be3fbeba988fb77d01c9857a422f7bc87cf40580a915cb3e434b49f43ad47f55"} Jul 15 12:08:50.624944 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:50.624870 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-76897f7847-gzs6x" Jul 15 12:08:50.626204 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:50.626172 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55594846d9-5mxfn" event={"ID":"a5d9e822-bb8b-44e5-889a-322898e09eea","Type":"ContainerStarted","Data":"15b665b7c4f1ee99c66157061d4e9c887b223a8d0e4a480271cb1624e68f8b88"} Jul 15 12:08:50.626319 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:50.626303 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55594846d9-5mxfn" Jul 15 12:08:50.660126 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:50.660086 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-76897f7847-gzs6x" podStartSLOduration=1.036752913 podStartE2EDuration="3.660072323s" totalImagesPullingTime="2.62331941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:50.659390532 +0000 UTC m=+598.072918962" watchObservedRunningTime="2026-07-15 12:08:50.660072323 +0000 UTC m=+598.073600699" Jul 15 12:08:50.685197 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:50.685138 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55594846d9-5mxfn" podStartSLOduration=1.1651553510000001 podStartE2EDuration="3.685119514s" totalImagesPullingTime="2.519964163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:50.68158601 +0000 UTC m=+598.095114371" watchObservedRunningTime="2026-07-15 12:08:50.685119514 +0000 UTC m=+598.098647890" Jul 15 12:08:53.059658 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:53.059630 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:08:53.064185 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:53.064165 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:08:59.707764 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:59.707720 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:59.857598 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:08:59.857528 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:01.634719 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:01.634687 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-76897f7847-gzs6x" Jul 15 12:09:01.635103 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:01.634778 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55594846d9-5mxfn" Jul 15 12:09:01.710325 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:01.710288 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76897f7847-gzs6x"] Jul 15 12:09:01.710562 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:01.710517 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-76897f7847-gzs6x" podUID="72198a09-0f56-4462-9258-e11ec9bdec0f" containerName="manager" containerID="cri-o://be3fbeba988fb77d01c9857a422f7bc87cf40580a915cb3e434b49f43ad47f55" gracePeriod=30 Jul 15 12:09:01.961150 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:01.961125 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76897f7847-gzs6x" Jul 15 12:09:02.070972 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:02.070938 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72198a09-0f56-4462-9258-e11ec9bdec0f-webhook-cert\") pod \"72198a09-0f56-4462-9258-e11ec9bdec0f\" (UID: \"72198a09-0f56-4462-9258-e11ec9bdec0f\") " Jul 15 12:09:02.071154 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:02.071029 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9qqw\" (UniqueName: \"kubernetes.io/projected/72198a09-0f56-4462-9258-e11ec9bdec0f-kube-api-access-d9qqw\") pod \"72198a09-0f56-4462-9258-e11ec9bdec0f\" (UID: \"72198a09-0f56-4462-9258-e11ec9bdec0f\") " Jul 15 12:09:02.073301 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:02.073266 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72198a09-0f56-4462-9258-e11ec9bdec0f-kube-api-access-d9qqw" (OuterVolumeSpecName: "kube-api-access-d9qqw") pod "72198a09-0f56-4462-9258-e11ec9bdec0f" (UID: "72198a09-0f56-4462-9258-e11ec9bdec0f"). InnerVolumeSpecName "kube-api-access-d9qqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:09:02.073301 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:02.073296 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72198a09-0f56-4462-9258-e11ec9bdec0f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "72198a09-0f56-4462-9258-e11ec9bdec0f" (UID: "72198a09-0f56-4462-9258-e11ec9bdec0f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:09:02.172525 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:02.172483 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9qqw\" (UniqueName: \"kubernetes.io/projected/72198a09-0f56-4462-9258-e11ec9bdec0f-kube-api-access-d9qqw\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:09:02.172525 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:02.172521 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72198a09-0f56-4462-9258-e11ec9bdec0f-webhook-cert\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:09:02.682981 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:02.682945 2570 generic.go:356] "Generic (PLEG): container finished" podID="72198a09-0f56-4462-9258-e11ec9bdec0f" containerID="be3fbeba988fb77d01c9857a422f7bc87cf40580a915cb3e434b49f43ad47f55" exitCode=0 Jul 15 12:09:02.683440 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:02.682990 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-76897f7847-gzs6x" Jul 15 12:09:02.683440 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:02.683027 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76897f7847-gzs6x" event={"ID":"72198a09-0f56-4462-9258-e11ec9bdec0f","Type":"ContainerDied","Data":"be3fbeba988fb77d01c9857a422f7bc87cf40580a915cb3e434b49f43ad47f55"} Jul 15 12:09:02.683440 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:02.683084 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-76897f7847-gzs6x" event={"ID":"72198a09-0f56-4462-9258-e11ec9bdec0f","Type":"ContainerDied","Data":"a200c3c48447030903293237df08694aa8d8fe7c78e532466b0dd21123c4f825"} Jul 15 12:09:02.683440 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:02.683107 2570 scope.go:117] "RemoveContainer" containerID="be3fbeba988fb77d01c9857a422f7bc87cf40580a915cb3e434b49f43ad47f55" Jul 15 12:09:02.692316 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:02.692297 2570 scope.go:117] "RemoveContainer" containerID="be3fbeba988fb77d01c9857a422f7bc87cf40580a915cb3e434b49f43ad47f55" Jul 15 12:09:02.692588 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:09:02.692566 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3fbeba988fb77d01c9857a422f7bc87cf40580a915cb3e434b49f43ad47f55\": container with ID starting with be3fbeba988fb77d01c9857a422f7bc87cf40580a915cb3e434b49f43ad47f55 not found: ID does not exist" containerID="be3fbeba988fb77d01c9857a422f7bc87cf40580a915cb3e434b49f43ad47f55" Jul 15 12:09:02.692652 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:02.692596 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3fbeba988fb77d01c9857a422f7bc87cf40580a915cb3e434b49f43ad47f55"} err="failed to get container status \"be3fbeba988fb77d01c9857a422f7bc87cf40580a915cb3e434b49f43ad47f55\": rpc error: code = NotFound desc = could not find container \"be3fbeba988fb77d01c9857a422f7bc87cf40580a915cb3e434b49f43ad47f55\": container with ID starting with be3fbeba988fb77d01c9857a422f7bc87cf40580a915cb3e434b49f43ad47f55 not found: ID does not exist" Jul 15 12:09:02.707719 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:02.707668 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-76897f7847-gzs6x"] Jul 15 12:09:02.709340 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:02.709317 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-76897f7847-gzs6x"] Jul 15 12:09:03.092964 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:03.092886 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72198a09-0f56-4462-9258-e11ec9bdec0f" path="/var/lib/kubelet/pods/72198a09-0f56-4462-9258-e11ec9bdec0f/volumes" Jul 15 12:09:23.183187 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.183105 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:23.282747 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.282709 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:23.666654 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.666618 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:23.743222 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.743191 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v"] Jul 15 12:09:23.743561 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.743549 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72198a09-0f56-4462-9258-e11ec9bdec0f" containerName="manager" Jul 15 12:09:23.743604 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.743563 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="72198a09-0f56-4462-9258-e11ec9bdec0f" containerName="manager" Jul 15 12:09:23.743637 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.743627 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="72198a09-0f56-4462-9258-e11ec9bdec0f" containerName="manager" Jul 15 12:09:23.748061 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.748038 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" Jul 15 12:09:23.751190 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.751171 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 15 12:09:23.751433 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.751194 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hgh4d\"" Jul 15 12:09:23.756902 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.756876 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v"] Jul 15 12:09:23.802593 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.802557 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:23.869096 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.869066 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77mds\" (UniqueName: \"kubernetes.io/projected/e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa-kube-api-access-77mds\") pod \"maas-api-66c7d954f5-qfs4v\" (UID: \"e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa\") " pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" Jul 15 12:09:23.869263 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.869111 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa-openshift-service-ca\") pod \"maas-api-66c7d954f5-qfs4v\" (UID: \"e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa\") " pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" Jul 15 12:09:23.869263 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.869138 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa-maas-api-tls\") pod \"maas-api-66c7d954f5-qfs4v\" (UID: \"e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa\") " pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" Jul 15 12:09:23.969977 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.969897 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77mds\" (UniqueName: \"kubernetes.io/projected/e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa-kube-api-access-77mds\") pod \"maas-api-66c7d954f5-qfs4v\" (UID: \"e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa\") " pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" Jul 15 12:09:23.969977 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.969944 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa-openshift-service-ca\") pod \"maas-api-66c7d954f5-qfs4v\" (UID: \"e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa\") " pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" Jul 15 12:09:23.969977 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.969974 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa-maas-api-tls\") pod \"maas-api-66c7d954f5-qfs4v\" (UID: \"e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa\") " pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" Jul 15 12:09:23.970544 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.970520 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa-openshift-service-ca\") pod \"maas-api-66c7d954f5-qfs4v\" (UID: \"e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa\") " pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" Jul 15 12:09:23.972445 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.972424 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa-maas-api-tls\") pod \"maas-api-66c7d954f5-qfs4v\" (UID: \"e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa\") " pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" Jul 15 12:09:23.977773 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:23.977756 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77mds\" (UniqueName: \"kubernetes.io/projected/e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa-kube-api-access-77mds\") pod \"maas-api-66c7d954f5-qfs4v\" (UID: \"e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa\") " pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" Jul 15 12:09:24.058714 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:24.058682 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" Jul 15 12:09:24.201314 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:24.201284 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v"] Jul 15 12:09:24.204331 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:09:24.204304 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cc2ed9_7abc_4bcf_8433_63a3891dd6fa.slice/crio-631b6c51cf80cc7e1372d35ba6a7d01d06bca47f0b60ec377da41860af9af62b WatchSource:0}: Error finding container 631b6c51cf80cc7e1372d35ba6a7d01d06bca47f0b60ec377da41860af9af62b: Status 404 returned error can't find the container with id 631b6c51cf80cc7e1372d35ba6a7d01d06bca47f0b60ec377da41860af9af62b Jul 15 12:09:24.265102 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:24.265011 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:24.380053 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:24.380023 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:24.782831 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:24.782782 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" event={"ID":"e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa","Type":"ContainerStarted","Data":"631b6c51cf80cc7e1372d35ba6a7d01d06bca47f0b60ec377da41860af9af62b"} Jul 15 12:09:26.793007 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:26.792974 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" event={"ID":"e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa","Type":"ContainerStarted","Data":"45fd3951544177ce5053141cf034204527ff9d619f4b063b810806fd4782b6d5"} Jul 15 12:09:26.793360 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:26.793044 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" Jul 15 12:09:32.804093 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:32.804064 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" Jul 15 12:09:32.822669 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:32.822621 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66c7d954f5-qfs4v" podStartSLOduration=8.25791553 podStartE2EDuration="9.822601875s" totalImagesPullingTime="1.564686345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:26.817161181 +0000 UTC m=+634.230689523" watchObservedRunningTime="2026-07-15 12:09:32.822601875 +0000 UTC m=+640.236130226" Jul 15 12:09:33.876935 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:33.876898 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:33.976879 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:33.976842 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:45.738164 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.738131 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng"] Jul 15 12:09:45.741339 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.741319 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.743676 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.743653 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 15 12:09:45.743676 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.743663 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 12:09:45.743862 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.743727 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 12:09:45.743862 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.743668 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hz8rk\"" Jul 15 12:09:45.755072 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.755052 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng"] Jul 15 12:09:45.827568 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.827529 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:45.866392 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.866360 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.866392 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.866392 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.866554 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.866415 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl59v\" (UniqueName: \"kubernetes.io/projected/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-kube-api-access-sl59v\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.866554 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.866509 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.866622 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.866572 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.866622 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.866600 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.967748 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.967711 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.967956 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.967789 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.967956 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.967822 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.967956 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.967843 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.968137 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.967994 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl59v\" (UniqueName: \"kubernetes.io/projected/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-kube-api-access-sl59v\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.968137 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.968049 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.968243 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.968144 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.968243 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.968186 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.968243 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.968218 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.970225 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.970203 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.970390 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.970372 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:45.975335 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:45.975317 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl59v\" (UniqueName: \"kubernetes.io/projected/4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100-kube-api-access-sl59v\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng\" (UID: \"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:46.052284 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:46.052220 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:46.190254 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:46.190220 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng"] Jul 15 12:09:46.193401 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:09:46.193376 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5d866e_fb8c_4a7c_8dfc_fa5982ab5100.slice/crio-895d20333b8b03ef630e1d98e2ae0174bd06077eb7e35d8a0386b915eecc679c WatchSource:0}: Error finding container 895d20333b8b03ef630e1d98e2ae0174bd06077eb7e35d8a0386b915eecc679c: Status 404 returned error can't find the container with id 895d20333b8b03ef630e1d98e2ae0174bd06077eb7e35d8a0386b915eecc679c Jul 15 12:09:46.888604 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:46.888569 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" event={"ID":"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100","Type":"ContainerStarted","Data":"895d20333b8b03ef630e1d98e2ae0174bd06077eb7e35d8a0386b915eecc679c"} Jul 15 12:09:47.894371 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:47.894332 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" event={"ID":"4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100","Type":"ContainerStarted","Data":"a52a1479cf3687c997a997f0cf6c74b90b8d1cefcb31700bfa3d446302098501"} Jul 15 12:09:47.912536 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:47.912487 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" podStartSLOduration=1.810172407 podStartE2EDuration="2.912473576s" totalImagesPullingTime="1.102301169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:47.911139105 +0000 UTC m=+655.324667457" watchObservedRunningTime="2026-07-15 12:09:47.912473576 +0000 UTC m=+655.326001927" Jul 15 12:09:51.807302 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:51.807266 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:51.876586 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:51.876549 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:56.052913 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:56.052877 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:56.053432 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:56.052927 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:56.065647 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:56.065623 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:09:56.944912 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:09:56.944885 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng" Jul 15 12:10:19.990766 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:10:19.990708 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:10:28.277987 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:10:28.277949 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:10:51.677539 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:10:51.677447 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:10:55.672511 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:10:55.672465 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:05.436556 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:11:05.436509 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:05.537027 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:11:05.536990 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:07.551272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:11:07.551234 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:51.179928 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:11:51.179881 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:53.436731 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:11:53.436696 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:02.826242 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:02.826203 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f894584f6-vnc5l"] Jul 15 12:12:02.829520 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:02.829498 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f894584f6-vnc5l" Jul 15 12:12:02.831694 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:02.831672 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 15 12:12:02.832072 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:02.832055 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-878tq\"" Jul 15 12:12:02.840911 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:02.840892 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f894584f6-vnc5l"] Jul 15 12:12:02.897121 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:02.897089 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6d613f76-23d4-4392-a2e2-f8658440a072-tls-cert\") pod \"authorino-7f894584f6-vnc5l\" (UID: \"6d613f76-23d4-4392-a2e2-f8658440a072\") " pod="kuadrant-system/authorino-7f894584f6-vnc5l" Jul 15 12:12:02.897121 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:02.897127 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5fd4\" (UniqueName: \"kubernetes.io/projected/6d613f76-23d4-4392-a2e2-f8658440a072-kube-api-access-f5fd4\") pod \"authorino-7f894584f6-vnc5l\" (UID: \"6d613f76-23d4-4392-a2e2-f8658440a072\") " pod="kuadrant-system/authorino-7f894584f6-vnc5l" Jul 15 12:12:02.998408 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:02.998366 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6d613f76-23d4-4392-a2e2-f8658440a072-tls-cert\") pod \"authorino-7f894584f6-vnc5l\" (UID: \"6d613f76-23d4-4392-a2e2-f8658440a072\") " pod="kuadrant-system/authorino-7f894584f6-vnc5l" Jul 15 12:12:02.998408 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:02.998408 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5fd4\" (UniqueName: \"kubernetes.io/projected/6d613f76-23d4-4392-a2e2-f8658440a072-kube-api-access-f5fd4\") pod \"authorino-7f894584f6-vnc5l\" (UID: \"6d613f76-23d4-4392-a2e2-f8658440a072\") " pod="kuadrant-system/authorino-7f894584f6-vnc5l" Jul 15 12:12:03.000983 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:03.000965 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6d613f76-23d4-4392-a2e2-f8658440a072-tls-cert\") pod \"authorino-7f894584f6-vnc5l\" (UID: \"6d613f76-23d4-4392-a2e2-f8658440a072\") " pod="kuadrant-system/authorino-7f894584f6-vnc5l" Jul 15 12:12:03.007155 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:03.007129 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5fd4\" (UniqueName: \"kubernetes.io/projected/6d613f76-23d4-4392-a2e2-f8658440a072-kube-api-access-f5fd4\") pod \"authorino-7f894584f6-vnc5l\" (UID: \"6d613f76-23d4-4392-a2e2-f8658440a072\") " pod="kuadrant-system/authorino-7f894584f6-vnc5l" Jul 15 12:12:03.138699 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:03.138664 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f894584f6-vnc5l" Jul 15 12:12:03.202271 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:03.202218 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:03.283319 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:03.283293 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f894584f6-vnc5l"] Jul 15 12:12:03.285039 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:12:03.285013 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d613f76_23d4_4392_a2e2_f8658440a072.slice/crio-ee4d13f82ac00e2422bffb5b1128da9d24e22b4cd9cb2d9179633a67f893c7bf WatchSource:0}: Error finding container ee4d13f82ac00e2422bffb5b1128da9d24e22b4cd9cb2d9179633a67f893c7bf: Status 404 returned error can't find the container with id ee4d13f82ac00e2422bffb5b1128da9d24e22b4cd9cb2d9179633a67f893c7bf Jul 15 12:12:03.286524 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:03.286507 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:12:03.290220 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:03.290197 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:03.502005 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:03.501930 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f894584f6-vnc5l" event={"ID":"6d613f76-23d4-4392-a2e2-f8658440a072","Type":"ContainerStarted","Data":"ee4d13f82ac00e2422bffb5b1128da9d24e22b4cd9cb2d9179633a67f893c7bf"} Jul 15 12:12:04.392120 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:04.391999 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:04.508751 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:04.508701 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f894584f6-vnc5l" event={"ID":"6d613f76-23d4-4392-a2e2-f8658440a072","Type":"ContainerStarted","Data":"eec4813ded7eed2d1001a7dcbe886610a45e05c7ed5437ed33329a5426809f16"} Jul 15 12:12:04.534720 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:04.534668 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7f894584f6-vnc5l" podStartSLOduration=1.685087915 podStartE2EDuration="2.53465083s" totalImagesPullingTime="849.562915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:12:04.532999338 +0000 UTC m=+791.946527689" watchObservedRunningTime="2026-07-15 12:12:04.53465083 +0000 UTC m=+791.948179180" Jul 15 12:12:04.606276 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:12:04.606245 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:13:25.545093 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:25.545059 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:13:25.637156 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:25.637120 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55594846d9-5mxfn"] Jul 15 12:13:25.637349 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:25.637324 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-55594846d9-5mxfn" podUID="a5d9e822-bb8b-44e5-889a-322898e09eea" containerName="manager" containerID="cri-o://15b665b7c4f1ee99c66157061d4e9c887b223a8d0e4a480271cb1624e68f8b88" gracePeriod=30 Jul 15 12:13:25.890705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:25.890679 2570 generic.go:356] "Generic (PLEG): container finished" podID="a5d9e822-bb8b-44e5-889a-322898e09eea" containerID="15b665b7c4f1ee99c66157061d4e9c887b223a8d0e4a480271cb1624e68f8b88" exitCode=0 Jul 15 12:13:25.890817 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:25.890734 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55594846d9-5mxfn" event={"ID":"a5d9e822-bb8b-44e5-889a-322898e09eea","Type":"ContainerDied","Data":"15b665b7c4f1ee99c66157061d4e9c887b223a8d0e4a480271cb1624e68f8b88"} Jul 15 12:13:25.890817 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:25.890793 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55594846d9-5mxfn" event={"ID":"a5d9e822-bb8b-44e5-889a-322898e09eea","Type":"ContainerDied","Data":"d42feb7a49f8b1562aaa61efec37f233235da6f7c6f7686ec50877e35cc7d752"} Jul 15 12:13:25.890817 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:25.890809 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d42feb7a49f8b1562aaa61efec37f233235da6f7c6f7686ec50877e35cc7d752" Jul 15 12:13:25.891669 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:25.891652 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55594846d9-5mxfn" Jul 15 12:13:25.951240 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:25.951203 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2lh6\" (UniqueName: \"kubernetes.io/projected/a5d9e822-bb8b-44e5-889a-322898e09eea-kube-api-access-x2lh6\") pod \"a5d9e822-bb8b-44e5-889a-322898e09eea\" (UID: \"a5d9e822-bb8b-44e5-889a-322898e09eea\") " Jul 15 12:13:25.951400 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:25.951261 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d9e822-bb8b-44e5-889a-322898e09eea-webhook-cert\") pod \"a5d9e822-bb8b-44e5-889a-322898e09eea\" (UID: \"a5d9e822-bb8b-44e5-889a-322898e09eea\") " Jul 15 12:13:25.953429 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:25.953398 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5d9e822-bb8b-44e5-889a-322898e09eea-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a5d9e822-bb8b-44e5-889a-322898e09eea" (UID: "a5d9e822-bb8b-44e5-889a-322898e09eea"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:13:25.953543 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:25.953452 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d9e822-bb8b-44e5-889a-322898e09eea-kube-api-access-x2lh6" (OuterVolumeSpecName: "kube-api-access-x2lh6") pod "a5d9e822-bb8b-44e5-889a-322898e09eea" (UID: "a5d9e822-bb8b-44e5-889a-322898e09eea"). InnerVolumeSpecName "kube-api-access-x2lh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:13:26.052302 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:26.052253 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2lh6\" (UniqueName: \"kubernetes.io/projected/a5d9e822-bb8b-44e5-889a-322898e09eea-kube-api-access-x2lh6\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:13:26.052302 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:26.052297 2570 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d9e822-bb8b-44e5-889a-322898e09eea-webhook-cert\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:13:26.894724 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:26.894694 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55594846d9-5mxfn" Jul 15 12:13:26.914444 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:26.914419 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55594846d9-5mxfn"] Jul 15 12:13:26.918372 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:26.918348 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55594846d9-5mxfn"] Jul 15 12:13:27.091902 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.091861 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5d9e822-bb8b-44e5-889a-322898e09eea" path="/var/lib/kubelet/pods/a5d9e822-bb8b-44e5-889a-322898e09eea/volumes" Jul 15 12:13:27.594131 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.594098 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:13:27.718800 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.718764 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55594846d9-q2qsp"] Jul 15 12:13:27.719181 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.719165 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d9e822-bb8b-44e5-889a-322898e09eea" containerName="manager" Jul 15 12:13:27.719272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.719184 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d9e822-bb8b-44e5-889a-322898e09eea" containerName="manager" Jul 15 12:13:27.719272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.719268 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5d9e822-bb8b-44e5-889a-322898e09eea" containerName="manager" Jul 15 12:13:27.722340 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.722316 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55594846d9-q2qsp" Jul 15 12:13:27.724492 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.724469 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 15 12:13:27.724612 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.724503 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 15 12:13:27.724612 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.724581 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wsx7r\"" Jul 15 12:13:27.732382 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.732361 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55594846d9-q2qsp"] Jul 15 12:13:27.870211 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.870180 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmblk\" (UniqueName: \"kubernetes.io/projected/4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02-kube-api-access-fmblk\") pod \"maas-controller-55594846d9-q2qsp\" (UID: \"4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02\") " pod="opendatahub/maas-controller-55594846d9-q2qsp" Jul 15 12:13:27.870361 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.870229 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02-webhook-cert\") pod \"maas-controller-55594846d9-q2qsp\" (UID: \"4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02\") " pod="opendatahub/maas-controller-55594846d9-q2qsp" Jul 15 12:13:27.970991 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.970958 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmblk\" (UniqueName: \"kubernetes.io/projected/4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02-kube-api-access-fmblk\") pod \"maas-controller-55594846d9-q2qsp\" (UID: \"4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02\") " pod="opendatahub/maas-controller-55594846d9-q2qsp" Jul 15 12:13:27.971376 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.971010 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02-webhook-cert\") pod \"maas-controller-55594846d9-q2qsp\" (UID: \"4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02\") " pod="opendatahub/maas-controller-55594846d9-q2qsp" Jul 15 12:13:27.973557 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.973538 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02-webhook-cert\") pod \"maas-controller-55594846d9-q2qsp\" (UID: \"4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02\") " pod="opendatahub/maas-controller-55594846d9-q2qsp" Jul 15 12:13:27.979132 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:27.979105 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmblk\" (UniqueName: \"kubernetes.io/projected/4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02-kube-api-access-fmblk\") pod \"maas-controller-55594846d9-q2qsp\" (UID: \"4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02\") " pod="opendatahub/maas-controller-55594846d9-q2qsp" Jul 15 12:13:28.032319 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:28.032277 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55594846d9-q2qsp" Jul 15 12:13:28.165506 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:28.165484 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55594846d9-q2qsp"] Jul 15 12:13:28.167622 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:13:28.167596 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adc5ad8_bd3d_420d_9e65_a7b7d4b47f02.slice/crio-031434405b28f066745b6bbfa8a277870fad29a067afc80545adefafc6929f87 WatchSource:0}: Error finding container 031434405b28f066745b6bbfa8a277870fad29a067afc80545adefafc6929f87: Status 404 returned error can't find the container with id 031434405b28f066745b6bbfa8a277870fad29a067afc80545adefafc6929f87 Jul 15 12:13:28.905380 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:28.905345 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55594846d9-q2qsp" event={"ID":"4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02","Type":"ContainerStarted","Data":"4a9f4feea80d95286f1b188c81e704b4698c2b5d0bf0a9a9550c7bd3acbfdc22"} Jul 15 12:13:28.905564 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:28.905405 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55594846d9-q2qsp" event={"ID":"4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02","Type":"ContainerStarted","Data":"031434405b28f066745b6bbfa8a277870fad29a067afc80545adefafc6929f87"} Jul 15 12:13:28.905564 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:28.905451 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55594846d9-q2qsp" Jul 15 12:13:28.925062 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:28.925010 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55594846d9-q2qsp" podStartSLOduration=1.339434711 podStartE2EDuration="1.924994858s" totalImagesPullingTime="585.560147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:13:28.923232757 +0000 UTC m=+876.336761105" watchObservedRunningTime="2026-07-15 12:13:28.924994858 +0000 UTC m=+876.338523208" Jul 15 12:13:39.912380 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:39.912350 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55594846d9-q2qsp" Jul 15 12:13:53.095883 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:53.095797 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:13:53.097923 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:13:53.097895 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:14:08.552200 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:14:08.552153 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:08.650233 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:14:08.650191 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:08.819777 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:14:08.819685 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:15.470525 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:14:15.470482 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:15.577630 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:14:15.577597 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:15.671339 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:14:15.671305 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:15.836929 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:14:15.836822 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:17.483239 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:14:17.483193 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:21.724599 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:14:21.724555 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:21.791442 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:14:21.791407 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:53.119592 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:14:53.119560 2570 scope.go:117] "RemoveContainer" containerID="15b665b7c4f1ee99c66157061d4e9c887b223a8d0e4a480271cb1624e68f8b88" Jul 15 12:15:00.140045 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:15:00.140009 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735295-b9xql"] Jul 15 12:15:00.144568 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:15:00.144551 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735295-b9xql" Jul 15 12:15:00.146572 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:15:00.146552 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tf4tp\"" Jul 15 12:15:00.150844 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:15:00.150820 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735295-b9xql"] Jul 15 12:15:00.155427 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:15:00.155408 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735295-b9xql" Jul 15 12:15:00.288388 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:15:00.288334 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735295-b9xql"] Jul 15 12:15:00.290827 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:15:00.290796 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c52611_e70f_4c8e_b5b0_6f497ff66c0c.slice/crio-bc3244ae021e158c38de51b8fc5991ab13b6e0c4dd54b8ed77f932876995f1a0 WatchSource:0}: Error finding container bc3244ae021e158c38de51b8fc5991ab13b6e0c4dd54b8ed77f932876995f1a0: Status 404 returned error can't find the container with id bc3244ae021e158c38de51b8fc5991ab13b6e0c4dd54b8ed77f932876995f1a0 Jul 15 12:15:00.331188 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:15:00.331154 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735295-b9xql" event={"ID":"53c52611-e70f-4c8e-b5b0-6f497ff66c0c","Type":"ContainerStarted","Data":"bc3244ae021e158c38de51b8fc5991ab13b6e0c4dd54b8ed77f932876995f1a0"} Jul 15 12:15:03.351873 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:15:03.351831 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735295-b9xql" event={"ID":"53c52611-e70f-4c8e-b5b0-6f497ff66c0c","Type":"ContainerStarted","Data":"a36372ba4a4934e51a371db775c283ea87aab70e39397dc2ee9357d4116133d0"} Jul 15 12:15:03.369032 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:15:03.368985 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735295-b9xql" podStartSLOduration=1.225415411 podStartE2EDuration="3.368968298s" totalImagesPullingTime="2.143552887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:15:03.366330522 +0000 UTC m=+970.779858884" watchObservedRunningTime="2026-07-15 12:15:03.368968298 +0000 UTC m=+970.782496654" Jul 15 12:15:04.358028 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:15:04.357995 2570 generic.go:356] "Generic (PLEG): container finished" podID="53c52611-e70f-4c8e-b5b0-6f497ff66c0c" containerID="a36372ba4a4934e51a371db775c283ea87aab70e39397dc2ee9357d4116133d0" exitCode=0 Jul 15 12:15:04.358435 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:15:04.358090 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735295-b9xql" event={"ID":"53c52611-e70f-4c8e-b5b0-6f497ff66c0c","Type":"ContainerDied","Data":"a36372ba4a4934e51a371db775c283ea87aab70e39397dc2ee9357d4116133d0"} Jul 15 12:15:05.496017 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:15:05.495991 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735295-b9xql" Jul 15 12:15:06.368947 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:15:06.368910 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735295-b9xql" event={"ID":"53c52611-e70f-4c8e-b5b0-6f497ff66c0c","Type":"ContainerDied","Data":"bc3244ae021e158c38de51b8fc5991ab13b6e0c4dd54b8ed77f932876995f1a0"} Jul 15 12:15:06.368947 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:15:06.368944 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735295-b9xql" Jul 15 12:15:06.369151 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:15:06.368969 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc3244ae021e158c38de51b8fc5991ab13b6e0c4dd54b8ed77f932876995f1a0" Jul 15 12:17:54.299594 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:17:54.299559 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:55.984013 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:17:55.983974 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:53.124940 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:18:53.124908 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:18:53.125470 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:18:53.125119 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:19:30.827254 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:19:30.827219 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:30.889279 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:19:30.889249 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:33.969860 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:19:33.969824 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:20:25.907708 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:20:25.907671 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:53.184427 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:23:53.184395 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:23:53.187613 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:23:53.187590 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:24:51.161635 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:24:51.161600 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:24:52.464292 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:24:52.464242 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:02.338121 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:02.338087 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:02.439841 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:02.439797 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:02.765552 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:02.765512 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:08.647581 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:08.647543 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:08.749767 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:08.749711 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:08.855652 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:08.855622 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:08.954963 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:08.954878 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:10.050856 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.050820 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:10.140548 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.140516 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq"] Jul 15 12:25:10.140933 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.140919 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c52611-e70f-4c8e-b5b0-6f497ff66c0c" containerName="cleanup" Jul 15 12:25:10.140988 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.140934 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c52611-e70f-4c8e-b5b0-6f497ff66c0c" containerName="cleanup" Jul 15 12:25:10.141023 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.141011 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c52611-e70f-4c8e-b5b0-6f497ff66c0c" containerName="cleanup" Jul 15 12:25:10.143255 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.143235 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" Jul 15 12:25:10.143970 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.143947 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:10.145255 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.145237 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-548eb552-serving-cert\"" Jul 15 12:25:10.154477 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.154451 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq"] Jul 15 12:25:10.227983 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.227952 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:10.314164 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.314085 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016f553a-aaeb-48d6-afea-ace999447114-openshift-service-ca\") pod \"maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq\" (UID: \"016f553a-aaeb-48d6-afea-ace999447114\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" Jul 15 12:25:10.314164 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.314121 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz6qp\" (UniqueName: \"kubernetes.io/projected/016f553a-aaeb-48d6-afea-ace999447114-kube-api-access-tz6qp\") pod \"maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq\" (UID: \"016f553a-aaeb-48d6-afea-ace999447114\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" Jul 15 12:25:10.314336 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.314215 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f553a-aaeb-48d6-afea-ace999447114-maas-api-tls\") pod \"maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq\" (UID: \"016f553a-aaeb-48d6-afea-ace999447114\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" Jul 15 12:25:10.399324 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.399289 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:10.415529 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.415503 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016f553a-aaeb-48d6-afea-ace999447114-openshift-service-ca\") pod \"maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq\" (UID: \"016f553a-aaeb-48d6-afea-ace999447114\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" Jul 15 12:25:10.415671 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.415535 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz6qp\" (UniqueName: \"kubernetes.io/projected/016f553a-aaeb-48d6-afea-ace999447114-kube-api-access-tz6qp\") pod \"maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq\" (UID: \"016f553a-aaeb-48d6-afea-ace999447114\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" Jul 15 12:25:10.415671 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.415568 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f553a-aaeb-48d6-afea-ace999447114-maas-api-tls\") pod \"maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq\" (UID: \"016f553a-aaeb-48d6-afea-ace999447114\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" Jul 15 12:25:10.416197 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.416175 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016f553a-aaeb-48d6-afea-ace999447114-openshift-service-ca\") pod \"maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq\" (UID: \"016f553a-aaeb-48d6-afea-ace999447114\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" Jul 15 12:25:10.418166 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.418145 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f553a-aaeb-48d6-afea-ace999447114-maas-api-tls\") pod \"maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq\" (UID: \"016f553a-aaeb-48d6-afea-ace999447114\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" Jul 15 12:25:10.423263 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.423241 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz6qp\" (UniqueName: \"kubernetes.io/projected/016f553a-aaeb-48d6-afea-ace999447114-kube-api-access-tz6qp\") pod \"maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq\" (UID: \"016f553a-aaeb-48d6-afea-ace999447114\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" Jul 15 12:25:10.454376 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.454350 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" Jul 15 12:25:10.590530 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.587647 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq"] Jul 15 12:25:10.592058 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:25:10.592028 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016f553a_aaeb_48d6_afea_ace999447114.slice/crio-3911758a5aebe3b603c641b097fc5f76a8f4bf1b11be220dd91421c620e65b08 WatchSource:0}: Error finding container 3911758a5aebe3b603c641b097fc5f76a8f4bf1b11be220dd91421c620e65b08: Status 404 returned error can't find the container with id 3911758a5aebe3b603c641b097fc5f76a8f4bf1b11be220dd91421c620e65b08 Jul 15 12:25:10.593384 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.593365 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:25:10.876541 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.876502 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:10.980659 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:10.980627 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:11.203648 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:11.203568 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" event={"ID":"016f553a-aaeb-48d6-afea-ace999447114","Type":"ContainerStarted","Data":"3911758a5aebe3b603c641b097fc5f76a8f4bf1b11be220dd91421c620e65b08"} Jul 15 12:25:12.209577 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:12.209544 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" event={"ID":"016f553a-aaeb-48d6-afea-ace999447114","Type":"ContainerStarted","Data":"e7d635442379f9b74287dd5e59dfd7cb058c253a38917b04af14a6b678379571"} Jul 15 12:25:12.210006 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:12.209691 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" Jul 15 12:25:12.228353 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:12.228310 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" podStartSLOduration=1.598585806 podStartE2EDuration="2.228290739s" totalImagesPullingTime="629.704933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:25:12.227687617 +0000 UTC m=+1579.641215967" watchObservedRunningTime="2026-07-15 12:25:12.228290739 +0000 UTC m=+1579.641819089" Jul 15 12:25:12.844476 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:12.844444 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:15.083093 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:15.083060 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb"] Jul 15 12:25:15.086364 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:15.086341 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb" Jul 15 12:25:15.088208 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:15.088182 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tf4tp\"" Jul 15 12:25:15.095157 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:15.095115 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb"] Jul 15 12:25:15.262353 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:15.262316 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c094d1e-6c52-46fb-a0ee-9c99c7b48176-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-548eb552-rctwb\" (UID: \"9c094d1e-6c52-46fb-a0ee-9c99c7b48176\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb" Jul 15 12:25:15.363184 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:15.363154 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c094d1e-6c52-46fb-a0ee-9c99c7b48176-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-548eb552-rctwb\" (UID: \"9c094d1e-6c52-46fb-a0ee-9c99c7b48176\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb" Jul 15 12:25:15.363818 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:15.363800 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c094d1e-6c52-46fb-a0ee-9c99c7b48176-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-548eb552-rctwb\" (UID: \"9c094d1e-6c52-46fb-a0ee-9c99c7b48176\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb" Jul 15 12:25:15.398205 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:15.398172 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb" Jul 15 12:25:15.537058 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:15.537027 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb"] Jul 15 12:25:15.539174 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:25:15.539144 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c094d1e_6c52_46fb_a0ee_9c99c7b48176.slice/crio-7ea391cf23cd2cb544b19327d4eb3e3bbcdfd67b3c6c441c606af967303a20d7 WatchSource:0}: Error finding container 7ea391cf23cd2cb544b19327d4eb3e3bbcdfd67b3c6c441c606af967303a20d7: Status 404 returned error can't find the container with id 7ea391cf23cd2cb544b19327d4eb3e3bbcdfd67b3c6c441c606af967303a20d7 Jul 15 12:25:16.229813 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:16.229777 2570 generic.go:356] "Generic (PLEG): container finished" podID="9c094d1e-6c52-46fb-a0ee-9c99c7b48176" containerID="3899e2919bcdabec965e4c50c27e6d8f98b54be7ec2de8606c5582151e41d2d6" exitCode=7 Jul 15 12:25:16.230198 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:16.229851 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb" event={"ID":"9c094d1e-6c52-46fb-a0ee-9c99c7b48176","Type":"ContainerDied","Data":"3899e2919bcdabec965e4c50c27e6d8f98b54be7ec2de8606c5582151e41d2d6"} Jul 15 12:25:16.230198 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:16.229900 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb" event={"ID":"9c094d1e-6c52-46fb-a0ee-9c99c7b48176","Type":"ContainerStarted","Data":"7ea391cf23cd2cb544b19327d4eb3e3bbcdfd67b3c6c441c606af967303a20d7"} Jul 15 12:25:16.230198 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:16.230017 2570 scope.go:117] "RemoveContainer" containerID="3899e2919bcdabec965e4c50c27e6d8f98b54be7ec2de8606c5582151e41d2d6" Jul 15 12:25:17.237524 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:17.237484 2570 generic.go:356] "Generic (PLEG): container finished" podID="9c094d1e-6c52-46fb-a0ee-9c99c7b48176" containerID="e17724d176d9f415fa460ebf1d15478a1d7a7b1a2e4f10e511f2b9bf8f876534" exitCode=7 Jul 15 12:25:17.237999 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:17.237571 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb" event={"ID":"9c094d1e-6c52-46fb-a0ee-9c99c7b48176","Type":"ContainerDied","Data":"e17724d176d9f415fa460ebf1d15478a1d7a7b1a2e4f10e511f2b9bf8f876534"} Jul 15 12:25:17.237999 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:17.237634 2570 scope.go:117] "RemoveContainer" containerID="3899e2919bcdabec965e4c50c27e6d8f98b54be7ec2de8606c5582151e41d2d6" Jul 15 12:25:17.237999 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:17.237859 2570 scope.go:117] "RemoveContainer" containerID="e17724d176d9f415fa460ebf1d15478a1d7a7b1a2e4f10e511f2b9bf8f876534" Jul 15 12:25:17.238170 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:25:17.238101 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-548eb552-rctwb_odh-ai-gateway-infra(9c094d1e-6c52-46fb-a0ee-9c99c7b48176)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb" podUID="9c094d1e-6c52-46fb-a0ee-9c99c7b48176" Jul 15 12:25:18.220351 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:18.220324 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" Jul 15 12:25:18.244304 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:18.244272 2570 scope.go:117] "RemoveContainer" containerID="e17724d176d9f415fa460ebf1d15478a1d7a7b1a2e4f10e511f2b9bf8f876534" Jul 15 12:25:18.244764 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:25:18.244536 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-548eb552-rctwb_odh-ai-gateway-infra(9c094d1e-6c52-46fb-a0ee-9c99c7b48176)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb" podUID="9c094d1e-6c52-46fb-a0ee-9c99c7b48176" Jul 15 12:25:32.086148 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:32.086119 2570 scope.go:117] "RemoveContainer" containerID="e17724d176d9f415fa460ebf1d15478a1d7a7b1a2e4f10e511f2b9bf8f876534" Jul 15 12:25:33.313514 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:33.313480 2570 generic.go:356] "Generic (PLEG): container finished" podID="9c094d1e-6c52-46fb-a0ee-9c99c7b48176" containerID="bede73d7fa1e34fcba3b90c1736801e8d942234d027ad01301ae49a73b6fe36a" exitCode=0 Jul 15 12:25:33.313949 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:33.313548 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb" event={"ID":"9c094d1e-6c52-46fb-a0ee-9c99c7b48176","Type":"ContainerDied","Data":"bede73d7fa1e34fcba3b90c1736801e8d942234d027ad01301ae49a73b6fe36a"} Jul 15 12:25:33.313949 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:33.313609 2570 scope.go:117] "RemoveContainer" containerID="e17724d176d9f415fa460ebf1d15478a1d7a7b1a2e4f10e511f2b9bf8f876534" Jul 15 12:25:34.347021 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:34.346989 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb"] Jul 15 12:25:34.450949 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:34.450927 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb" Jul 15 12:25:34.532718 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:34.532688 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c094d1e-6c52-46fb-a0ee-9c99c7b48176-maas-api-service-ca\") pod \"9c094d1e-6c52-46fb-a0ee-9c99c7b48176\" (UID: \"9c094d1e-6c52-46fb-a0ee-9c99c7b48176\") " Jul 15 12:25:34.533013 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:34.532994 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c094d1e-6c52-46fb-a0ee-9c99c7b48176-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c094d1e-6c52-46fb-a0ee-9c99c7b48176" (UID: "9c094d1e-6c52-46fb-a0ee-9c99c7b48176"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:25:34.634175 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:34.634148 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c094d1e-6c52-46fb-a0ee-9c99c7b48176-maas-api-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:25:35.326633 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:35.326603 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb" Jul 15 12:25:35.326814 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:35.326648 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb" event={"ID":"9c094d1e-6c52-46fb-a0ee-9c99c7b48176","Type":"ContainerDied","Data":"7ea391cf23cd2cb544b19327d4eb3e3bbcdfd67b3c6c441c606af967303a20d7"} Jul 15 12:25:35.326814 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:35.326699 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ea391cf23cd2cb544b19327d4eb3e3bbcdfd67b3c6c441c606af967303a20d7" Jul 15 12:25:35.354466 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:35.354394 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb"] Jul 15 12:25:35.359499 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:35.359480 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-rctwb"] Jul 15 12:25:37.092231 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:37.092201 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c094d1e-6c52-46fb-a0ee-9c99c7b48176" path="/var/lib/kubelet/pods/9c094d1e-6c52-46fb-a0ee-9c99c7b48176/volumes" Jul 15 12:25:42.402885 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:42.402851 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:44.737592 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:25:44.737512 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:15.132854 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:15.132819 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-8fjqr"] Jul 15 12:26:15.133303 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:15.133167 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c094d1e-6c52-46fb-a0ee-9c99c7b48176" containerName="revoke-keys" Jul 15 12:26:15.133303 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:15.133178 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c094d1e-6c52-46fb-a0ee-9c99c7b48176" containerName="revoke-keys" Jul 15 12:26:15.133303 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:15.133195 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c094d1e-6c52-46fb-a0ee-9c99c7b48176" containerName="revoke-keys" Jul 15 12:26:15.133303 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:15.133201 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c094d1e-6c52-46fb-a0ee-9c99c7b48176" containerName="revoke-keys" Jul 15 12:26:15.133303 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:15.133259 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c094d1e-6c52-46fb-a0ee-9c99c7b48176" containerName="revoke-keys" Jul 15 12:26:15.133303 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:15.133269 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c094d1e-6c52-46fb-a0ee-9c99c7b48176" containerName="revoke-keys" Jul 15 12:26:15.136340 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:15.136323 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-8fjqr" Jul 15 12:26:15.138146 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:15.138122 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tf4tp\"" Jul 15 12:26:15.144286 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:15.144261 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-8fjqr"] Jul 15 12:26:15.197829 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:15.197794 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5621d2-a481-43cf-a0cb-a787b2f9bc7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-548eb552-8fjqr\" (UID: \"7b5621d2-a481-43cf-a0cb-a787b2f9bc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-8fjqr" Jul 15 12:26:15.298558 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:15.298522 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5621d2-a481-43cf-a0cb-a787b2f9bc7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-548eb552-8fjqr\" (UID: \"7b5621d2-a481-43cf-a0cb-a787b2f9bc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-8fjqr" Jul 15 12:26:15.299146 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:15.299126 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5621d2-a481-43cf-a0cb-a787b2f9bc7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-548eb552-8fjqr\" (UID: \"7b5621d2-a481-43cf-a0cb-a787b2f9bc7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-8fjqr" Jul 15 12:26:15.448488 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:15.448391 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-8fjqr" Jul 15 12:26:15.581240 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:15.581213 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-8fjqr"] Jul 15 12:26:15.586347 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:26:15.586287 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5621d2_a481_43cf_a0cb_a787b2f9bc7d.slice/crio-9d79d39a35f7cc696a5eaeab30b7299c0c6a6e96f4922baf2c30d2962e97eb5d WatchSource:0}: Error finding container 9d79d39a35f7cc696a5eaeab30b7299c0c6a6e96f4922baf2c30d2962e97eb5d: Status 404 returned error can't find the container with id 9d79d39a35f7cc696a5eaeab30b7299c0c6a6e96f4922baf2c30d2962e97eb5d Jul 15 12:26:16.512042 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:16.512010 2570 generic.go:356] "Generic (PLEG): container finished" podID="7b5621d2-a481-43cf-a0cb-a787b2f9bc7d" containerID="ada8fc0a79a623173819f18c80e88fb47fba68ba6adbe9dab20192cefb8643fb" exitCode=0 Jul 15 12:26:16.512455 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:16.512098 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-8fjqr" event={"ID":"7b5621d2-a481-43cf-a0cb-a787b2f9bc7d","Type":"ContainerDied","Data":"ada8fc0a79a623173819f18c80e88fb47fba68ba6adbe9dab20192cefb8643fb"} Jul 15 12:26:16.512455 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:16.512155 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-8fjqr" event={"ID":"7b5621d2-a481-43cf-a0cb-a787b2f9bc7d","Type":"ContainerStarted","Data":"9d79d39a35f7cc696a5eaeab30b7299c0c6a6e96f4922baf2c30d2962e97eb5d"} Jul 15 12:26:17.639579 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:17.639556 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-8fjqr" Jul 15 12:26:17.720302 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:17.720267 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5621d2-a481-43cf-a0cb-a787b2f9bc7d-maas-api-service-ca\") pod \"7b5621d2-a481-43cf-a0cb-a787b2f9bc7d\" (UID: \"7b5621d2-a481-43cf-a0cb-a787b2f9bc7d\") " Jul 15 12:26:17.720634 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:17.720611 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5621d2-a481-43cf-a0cb-a787b2f9bc7d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b5621d2-a481-43cf-a0cb-a787b2f9bc7d" (UID: "7b5621d2-a481-43cf-a0cb-a787b2f9bc7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:26:17.821811 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:17.821704 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5621d2-a481-43cf-a0cb-a787b2f9bc7d-maas-api-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:26:18.522473 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:18.522444 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-8fjqr" Jul 15 12:26:18.522656 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:18.522437 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-548eb552-8fjqr" event={"ID":"7b5621d2-a481-43cf-a0cb-a787b2f9bc7d","Type":"ContainerDied","Data":"9d79d39a35f7cc696a5eaeab30b7299c0c6a6e96f4922baf2c30d2962e97eb5d"} Jul 15 12:26:18.522656 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:18.522578 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d79d39a35f7cc696a5eaeab30b7299c0c6a6e96f4922baf2c30d2962e97eb5d" Jul 15 12:26:25.554918 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:25.554885 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq"] Jul 15 12:26:25.555455 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:25.555426 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" podUID="016f553a-aaeb-48d6-afea-ace999447114" containerName="maas-api" containerID="cri-o://e7d635442379f9b74287dd5e59dfd7cb058c253a38917b04af14a6b678379571" gracePeriod=30 Jul 15 12:26:25.591179 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:25.591147 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:25.803189 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:25.803163 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" Jul 15 12:26:25.895098 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:25.895067 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f553a-aaeb-48d6-afea-ace999447114-maas-api-tls\") pod \"016f553a-aaeb-48d6-afea-ace999447114\" (UID: \"016f553a-aaeb-48d6-afea-ace999447114\") " Jul 15 12:26:25.895292 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:25.895126 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016f553a-aaeb-48d6-afea-ace999447114-openshift-service-ca\") pod \"016f553a-aaeb-48d6-afea-ace999447114\" (UID: \"016f553a-aaeb-48d6-afea-ace999447114\") " Jul 15 12:26:25.895292 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:25.895176 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz6qp\" (UniqueName: \"kubernetes.io/projected/016f553a-aaeb-48d6-afea-ace999447114-kube-api-access-tz6qp\") pod \"016f553a-aaeb-48d6-afea-ace999447114\" (UID: \"016f553a-aaeb-48d6-afea-ace999447114\") " Jul 15 12:26:25.895472 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:25.895449 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016f553a-aaeb-48d6-afea-ace999447114-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "016f553a-aaeb-48d6-afea-ace999447114" (UID: "016f553a-aaeb-48d6-afea-ace999447114"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:26:25.897439 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:25.897413 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016f553a-aaeb-48d6-afea-ace999447114-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "016f553a-aaeb-48d6-afea-ace999447114" (UID: "016f553a-aaeb-48d6-afea-ace999447114"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:26:25.897551 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:25.897434 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016f553a-aaeb-48d6-afea-ace999447114-kube-api-access-tz6qp" (OuterVolumeSpecName: "kube-api-access-tz6qp") pod "016f553a-aaeb-48d6-afea-ace999447114" (UID: "016f553a-aaeb-48d6-afea-ace999447114"). InnerVolumeSpecName "kube-api-access-tz6qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:26:25.996693 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:25.996657 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016f553a-aaeb-48d6-afea-ace999447114-openshift-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:26:25.996693 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:25.996684 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz6qp\" (UniqueName: \"kubernetes.io/projected/016f553a-aaeb-48d6-afea-ace999447114-kube-api-access-tz6qp\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:26:25.996693 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:25.996694 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016f553a-aaeb-48d6-afea-ace999447114-maas-api-tls\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:26:26.560624 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:26.560591 2570 generic.go:356] "Generic (PLEG): container finished" podID="016f553a-aaeb-48d6-afea-ace999447114" containerID="e7d635442379f9b74287dd5e59dfd7cb058c253a38917b04af14a6b678379571" exitCode=0 Jul 15 12:26:26.561017 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:26.560638 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" Jul 15 12:26:26.561017 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:26.560677 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" event={"ID":"016f553a-aaeb-48d6-afea-ace999447114","Type":"ContainerDied","Data":"e7d635442379f9b74287dd5e59dfd7cb058c253a38917b04af14a6b678379571"} Jul 15 12:26:26.561017 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:26.560726 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq" event={"ID":"016f553a-aaeb-48d6-afea-ace999447114","Type":"ContainerDied","Data":"3911758a5aebe3b603c641b097fc5f76a8f4bf1b11be220dd91421c620e65b08"} Jul 15 12:26:26.561017 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:26.560772 2570 scope.go:117] "RemoveContainer" containerID="e7d635442379f9b74287dd5e59dfd7cb058c253a38917b04af14a6b678379571" Jul 15 12:26:26.569929 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:26.569915 2570 scope.go:117] "RemoveContainer" containerID="e7d635442379f9b74287dd5e59dfd7cb058c253a38917b04af14a6b678379571" Jul 15 12:26:26.570169 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:26:26.570149 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d635442379f9b74287dd5e59dfd7cb058c253a38917b04af14a6b678379571\": container with ID starting with e7d635442379f9b74287dd5e59dfd7cb058c253a38917b04af14a6b678379571 not found: ID does not exist" containerID="e7d635442379f9b74287dd5e59dfd7cb058c253a38917b04af14a6b678379571" Jul 15 12:26:26.570206 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:26.570178 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d635442379f9b74287dd5e59dfd7cb058c253a38917b04af14a6b678379571"} err="failed to get container status \"e7d635442379f9b74287dd5e59dfd7cb058c253a38917b04af14a6b678379571\": rpc error: code = NotFound desc = could not find container \"e7d635442379f9b74287dd5e59dfd7cb058c253a38917b04af14a6b678379571\": container with ID starting with e7d635442379f9b74287dd5e59dfd7cb058c253a38917b04af14a6b678379571 not found: ID does not exist" Jul 15 12:26:26.582989 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:26.582960 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq"] Jul 15 12:26:26.589027 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:26.589008 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-548eb552-768ffbb8d4-xhvdq"] Jul 15 12:26:27.092543 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:27.092501 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="016f553a-aaeb-48d6-afea-ace999447114" path="/var/lib/kubelet/pods/016f553a-aaeb-48d6-afea-ace999447114/volumes" Jul 15 12:26:30.224782 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:30.224731 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:36.296323 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:36.296292 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:36.379516 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:36.379479 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:40.420128 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.420091 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:40.809614 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.809514 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:40.901703 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.901668 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:40.936675 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.936643 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv"] Jul 15 12:26:40.937189 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.937174 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5621d2-a481-43cf-a0cb-a787b2f9bc7d" containerName="revoke-keys" Jul 15 12:26:40.937234 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.937194 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5621d2-a481-43cf-a0cb-a787b2f9bc7d" containerName="revoke-keys" Jul 15 12:26:40.937234 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.937213 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="016f553a-aaeb-48d6-afea-ace999447114" containerName="maas-api" Jul 15 12:26:40.937234 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.937221 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="016f553a-aaeb-48d6-afea-ace999447114" containerName="maas-api" Jul 15 12:26:40.937319 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.937237 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c094d1e-6c52-46fb-a0ee-9c99c7b48176" containerName="revoke-keys" Jul 15 12:26:40.937319 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.937246 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c094d1e-6c52-46fb-a0ee-9c99c7b48176" containerName="revoke-keys" Jul 15 12:26:40.937382 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.937346 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c094d1e-6c52-46fb-a0ee-9c99c7b48176" containerName="revoke-keys" Jul 15 12:26:40.937382 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.937358 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b5621d2-a481-43cf-a0cb-a787b2f9bc7d" containerName="revoke-keys" Jul 15 12:26:40.937382 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.937368 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="016f553a-aaeb-48d6-afea-ace999447114" containerName="maas-api" Jul 15 12:26:40.942286 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.942264 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:40.944417 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.944386 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 12:26:40.944551 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.944430 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-bbac7fd2-gw-openshift-default-dockercfg-58j9h\"" Jul 15 12:26:40.955639 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.955613 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv"] Jul 15 12:26:40.995238 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:40.995204 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:41.033795 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.033734 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-workload-certs\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.033795 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.033799 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51363008-5bed-4213-9550-f106a28fcb90-istiod-ca-cert\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.034007 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.033828 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkd2v\" (UniqueName: \"kubernetes.io/projected/51363008-5bed-4213-9550-f106a28fcb90-kube-api-access-wkd2v\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.034007 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.033852 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-workload-socket\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.034007 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.033971 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51363008-5bed-4213-9550-f106a28fcb90-istio-podinfo\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.034007 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.033998 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-istio-envoy\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.034156 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.034036 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-istio-data\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.034156 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.034059 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51363008-5bed-4213-9550-f106a28fcb90-istio-token\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.034156 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.034080 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-credential-socket\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.135114 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.135060 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51363008-5bed-4213-9550-f106a28fcb90-istio-podinfo\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.135114 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.135117 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-istio-envoy\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.135376 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.135163 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-istio-data\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.135376 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.135192 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51363008-5bed-4213-9550-f106a28fcb90-istio-token\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.135376 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.135217 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-credential-socket\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.135376 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.135274 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-workload-certs\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.135376 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.135297 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51363008-5bed-4213-9550-f106a28fcb90-istiod-ca-cert\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.135376 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.135326 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkd2v\" (UniqueName: \"kubernetes.io/projected/51363008-5bed-4213-9550-f106a28fcb90-kube-api-access-wkd2v\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.135376 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.135354 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-workload-socket\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.135713 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.135660 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-istio-data\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.135803 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.135713 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-credential-socket\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.135803 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.135782 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-workload-certs\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.136225 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.136209 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-workload-socket\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.136447 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.136425 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51363008-5bed-4213-9550-f106a28fcb90-istiod-ca-cert\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.137629 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.137610 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-istio-envoy\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.137862 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.137847 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51363008-5bed-4213-9550-f106a28fcb90-istio-podinfo\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.146327 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.146303 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51363008-5bed-4213-9550-f106a28fcb90-istio-token\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.146518 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.146494 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkd2v\" (UniqueName: \"kubernetes.io/projected/51363008-5bed-4213-9550-f106a28fcb90-kube-api-access-wkd2v\") pod \"e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.255684 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.255653 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:41.394800 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.394753 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv"] Jul 15 12:26:41.396990 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:26:41.396961 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51363008_5bed_4213_9550_f106a28fcb90.slice/crio-d0b4beffabe8ddcd9d37f7dd87a3c53f9bc926a0132cfdb3a9b6b553a284934d WatchSource:0}: Error finding container d0b4beffabe8ddcd9d37f7dd87a3c53f9bc926a0132cfdb3a9b6b553a284934d: Status 404 returned error can't find the container with id d0b4beffabe8ddcd9d37f7dd87a3c53f9bc926a0132cfdb3a9b6b553a284934d Jul 15 12:26:41.635956 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:41.635919 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" event={"ID":"51363008-5bed-4213-9550-f106a28fcb90","Type":"ContainerStarted","Data":"d0b4beffabe8ddcd9d37f7dd87a3c53f9bc926a0132cfdb3a9b6b553a284934d"} Jul 15 12:26:42.423358 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.423321 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:42.642788 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.642712 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt"] Jul 15 12:26:42.646596 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.646570 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" Jul 15 12:26:42.648997 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.648803 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-bbac7fd2-serving-cert\"" Jul 15 12:26:42.659411 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.659374 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt"] Jul 15 12:26:42.726412 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.726327 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:42.750457 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.750396 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-maas-api-tls\") pod \"maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt\" (UID: \"1b08d73a-fec3-44a7-89b5-9d53f0dd622f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" Jul 15 12:26:42.750615 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.750474 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5f9k\" (UniqueName: \"kubernetes.io/projected/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-kube-api-access-w5f9k\") pod \"maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt\" (UID: \"1b08d73a-fec3-44a7-89b5-9d53f0dd622f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" Jul 15 12:26:42.750615 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.750563 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-openshift-service-ca\") pod \"maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt\" (UID: \"1b08d73a-fec3-44a7-89b5-9d53f0dd622f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" Jul 15 12:26:42.851167 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.851138 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-openshift-service-ca\") pod \"maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt\" (UID: \"1b08d73a-fec3-44a7-89b5-9d53f0dd622f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" Jul 15 12:26:42.851338 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.851228 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-maas-api-tls\") pod \"maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt\" (UID: \"1b08d73a-fec3-44a7-89b5-9d53f0dd622f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" Jul 15 12:26:42.851338 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.851259 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5f9k\" (UniqueName: \"kubernetes.io/projected/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-kube-api-access-w5f9k\") pod \"maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt\" (UID: \"1b08d73a-fec3-44a7-89b5-9d53f0dd622f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" Jul 15 12:26:42.851886 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.851853 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-openshift-service-ca\") pod \"maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt\" (UID: \"1b08d73a-fec3-44a7-89b5-9d53f0dd622f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" Jul 15 12:26:42.853952 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.853926 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-maas-api-tls\") pod \"maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt\" (UID: \"1b08d73a-fec3-44a7-89b5-9d53f0dd622f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" Jul 15 12:26:42.859950 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.859930 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5f9k\" (UniqueName: \"kubernetes.io/projected/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-kube-api-access-w5f9k\") pod \"maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt\" (UID: \"1b08d73a-fec3-44a7-89b5-9d53f0dd622f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" Jul 15 12:26:42.906337 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.906297 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:42.960889 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:42.960857 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" Jul 15 12:26:43.192799 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:43.192721 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:43.302477 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:43.302417 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:26:43.718866 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:43.718831 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:26:43.719132 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:43.718900 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:26:43.719132 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:43.718937 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:26:43.787682 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:43.787656 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt"] Jul 15 12:26:43.789254 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:26:43.789218 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b08d73a_fec3_44a7_89b5_9d53f0dd622f.slice/crio-44cc7772f7f49da39edb578c1702b22dcf770e0e5d7424a0fd9fda5645508a42 WatchSource:0}: Error finding container 44cc7772f7f49da39edb578c1702b22dcf770e0e5d7424a0fd9fda5645508a42: Status 404 returned error can't find the container with id 44cc7772f7f49da39edb578c1702b22dcf770e0e5d7424a0fd9fda5645508a42 Jul 15 12:26:44.654033 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:44.654000 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" event={"ID":"51363008-5bed-4213-9550-f106a28fcb90","Type":"ContainerStarted","Data":"475ad35f337431218132905f66f2376f69ebae02471fb3662981b4a4dd32c3d8"} Jul 15 12:26:44.655511 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:44.655486 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" event={"ID":"1b08d73a-fec3-44a7-89b5-9d53f0dd622f","Type":"ContainerStarted","Data":"c79e78450371b28b652deb6921869da53a7a3609a8009c5a7c8d1713c46fb870"} Jul 15 12:26:44.655644 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:44.655536 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" event={"ID":"1b08d73a-fec3-44a7-89b5-9d53f0dd622f","Type":"ContainerStarted","Data":"44cc7772f7f49da39edb578c1702b22dcf770e0e5d7424a0fd9fda5645508a42"} Jul 15 12:26:44.655644 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:44.655577 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" Jul 15 12:26:44.679535 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:44.679485 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" podStartSLOduration=2.359928996 podStartE2EDuration="4.67946494s" totalImagesPullingTime="2.319535944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:26:44.678769278 +0000 UTC m=+1672.092297640" watchObservedRunningTime="2026-07-15 12:26:44.67946494 +0000 UTC m=+1672.092993291" Jul 15 12:26:44.695213 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:44.695165 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" podStartSLOduration=2.264592317 podStartE2EDuration="2.695146339s" totalImagesPullingTime="430.554022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:26:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:26:44.694276888 +0000 UTC m=+1672.107805237" watchObservedRunningTime="2026-07-15 12:26:44.695146339 +0000 UTC m=+1672.108674688" Jul 15 12:26:45.256484 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:45.256450 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:45.261134 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:45.261111 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:45.661390 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:45.661353 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:45.662222 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:45.662204 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:50.674036 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:50.674005 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" Jul 15 12:26:57.723574 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:57.723538 2570 generic.go:356] "Generic (PLEG): container finished" podID="51363008-5bed-4213-9550-f106a28fcb90" containerID="475ad35f337431218132905f66f2376f69ebae02471fb3662981b4a4dd32c3d8" exitCode=0 Jul 15 12:26:57.724066 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:57.723594 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" event={"ID":"51363008-5bed-4213-9550-f106a28fcb90","Type":"ContainerDied","Data":"475ad35f337431218132905f66f2376f69ebae02471fb3662981b4a4dd32c3d8"} Jul 15 12:26:57.724066 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:57.723940 2570 scope.go:117] "RemoveContainer" containerID="475ad35f337431218132905f66f2376f69ebae02471fb3662981b4a4dd32c3d8" Jul 15 12:26:57.724922 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:57.724890 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:26:57.724973 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:57.724964 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:26:57.725009 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:57.724994 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:26:58.730316 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:58.730282 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" event={"ID":"51363008-5bed-4213-9550-f106a28fcb90","Type":"ContainerStarted","Data":"7180aa41cbebf8331bcefd574647195d82d15183369143109f4b469deb16180e"} Jul 15 12:26:59.256463 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:59.256429 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:26:59.257588 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:59.257569 2570 patch_prober.go:28] interesting pod/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 15 12:26:59.257663 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:26:59.257612 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" podUID="51363008-5bed-4213-9550-f106a28fcb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 15 12:27:00.256190 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:00.256152 2570 patch_prober.go:28] interesting pod/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 15 12:27:00.256556 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:00.256204 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" podUID="51363008-5bed-4213-9550-f106a28fcb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 15 12:27:00.661593 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:00.661555 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:27:01.256119 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:01.256082 2570 patch_prober.go:28] interesting pod/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 15 12:27:01.256301 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:01.256136 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" podUID="51363008-5bed-4213-9550-f106a28fcb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 15 12:27:02.256196 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:02.256158 2570 patch_prober.go:28] interesting pod/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 15 12:27:02.256378 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:02.256223 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" podUID="51363008-5bed-4213-9550-f106a28fcb90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 15 12:27:03.259556 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:03.259524 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:27:03.260615 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:03.260595 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:27:46.793169 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:46.793133 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj"] Jul 15 12:27:46.796630 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:46.796608 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj" Jul 15 12:27:46.798610 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:46.798588 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tf4tp\"" Jul 15 12:27:46.804780 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:46.804716 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj"] Jul 15 12:27:46.841114 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:46.841086 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cfe7d47-396c-47ab-a37e-139ac9069afd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj\" (UID: \"5cfe7d47-396c-47ab-a37e-139ac9069afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj" Jul 15 12:27:46.871833 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:46.871803 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv"] Jul 15 12:27:46.872130 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:46.872081 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" podUID="51363008-5bed-4213-9550-f106a28fcb90" containerName="istio-proxy" containerID="cri-o://7180aa41cbebf8331bcefd574647195d82d15183369143109f4b469deb16180e" gracePeriod=30 Jul 15 12:27:46.899506 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:46.899476 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:46.941775 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:46.941726 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cfe7d47-396c-47ab-a37e-139ac9069afd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj\" (UID: \"5cfe7d47-396c-47ab-a37e-139ac9069afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj" Jul 15 12:27:46.942378 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:46.942354 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cfe7d47-396c-47ab-a37e-139ac9069afd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj\" (UID: \"5cfe7d47-396c-47ab-a37e-139ac9069afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj" Jul 15 12:27:47.003639 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:47.003611 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:47.109069 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:47.109041 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj" Jul 15 12:27:47.244687 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:47.244657 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj"] Jul 15 12:27:47.248237 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:27:47.248213 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfe7d47_396c_47ab_a37e_139ac9069afd.slice/crio-e83b93670c4fb82a8a99a5fb2d1f7bf2f065e2c84e4db08603c4d916441164f5 WatchSource:0}: Error finding container e83b93670c4fb82a8a99a5fb2d1f7bf2f065e2c84e4db08603c4d916441164f5: Status 404 returned error can't find the container with id e83b93670c4fb82a8a99a5fb2d1f7bf2f065e2c84e4db08603c4d916441164f5 Jul 15 12:27:47.974618 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:47.974581 2570 generic.go:356] "Generic (PLEG): container finished" podID="5cfe7d47-396c-47ab-a37e-139ac9069afd" containerID="0e4b23c9501789ef88d2bc80d262b64a832d6c020567ad4505fa7a121f09d03e" exitCode=0 Jul 15 12:27:47.974618 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:47.974616 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj" event={"ID":"5cfe7d47-396c-47ab-a37e-139ac9069afd","Type":"ContainerDied","Data":"0e4b23c9501789ef88d2bc80d262b64a832d6c020567ad4505fa7a121f09d03e"} Jul 15 12:27:47.975031 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:47.974656 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj" event={"ID":"5cfe7d47-396c-47ab-a37e-139ac9069afd","Type":"ContainerStarted","Data":"e83b93670c4fb82a8a99a5fb2d1f7bf2f065e2c84e4db08603c4d916441164f5"} Jul 15 12:27:48.262544 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:48.262459 2570 patch_prober.go:28] interesting pod/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 12:27:48.262544 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:48.262526 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" podUID="51363008-5bed-4213-9550-f106a28fcb90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 12:27:49.110875 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:49.110847 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj" Jul 15 12:27:49.164113 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:49.164082 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cfe7d47-396c-47ab-a37e-139ac9069afd-maas-api-service-ca\") pod \"5cfe7d47-396c-47ab-a37e-139ac9069afd\" (UID: \"5cfe7d47-396c-47ab-a37e-139ac9069afd\") " Jul 15 12:27:49.164495 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:49.164471 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cfe7d47-396c-47ab-a37e-139ac9069afd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5cfe7d47-396c-47ab-a37e-139ac9069afd" (UID: "5cfe7d47-396c-47ab-a37e-139ac9069afd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:27:49.265752 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:49.265659 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cfe7d47-396c-47ab-a37e-139ac9069afd-maas-api-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:27:49.985074 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:49.985037 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj" event={"ID":"5cfe7d47-396c-47ab-a37e-139ac9069afd","Type":"ContainerDied","Data":"e83b93670c4fb82a8a99a5fb2d1f7bf2f065e2c84e4db08603c4d916441164f5"} Jul 15 12:27:49.985074 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:49.985065 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj" Jul 15 12:27:49.985276 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:49.985087 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e83b93670c4fb82a8a99a5fb2d1f7bf2f065e2c84e4db08603c4d916441164f5" Jul 15 12:27:51.995872 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:51.995847 2570 generic.go:356] "Generic (PLEG): container finished" podID="51363008-5bed-4213-9550-f106a28fcb90" containerID="7180aa41cbebf8331bcefd574647195d82d15183369143109f4b469deb16180e" exitCode=0 Jul 15 12:27:51.996193 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:51.995934 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" event={"ID":"51363008-5bed-4213-9550-f106a28fcb90","Type":"ContainerDied","Data":"7180aa41cbebf8331bcefd574647195d82d15183369143109f4b469deb16180e"} Jul 15 12:27:51.996193 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:51.996011 2570 scope.go:117] "RemoveContainer" containerID="475ad35f337431218132905f66f2376f69ebae02471fb3662981b4a4dd32c3d8" Jul 15 12:27:52.122965 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.122940 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:27:52.192066 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.191980 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-workload-socket\") pod \"51363008-5bed-4213-9550-f106a28fcb90\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " Jul 15 12:27:52.192066 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.192020 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkd2v\" (UniqueName: \"kubernetes.io/projected/51363008-5bed-4213-9550-f106a28fcb90-kube-api-access-wkd2v\") pod \"51363008-5bed-4213-9550-f106a28fcb90\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " Jul 15 12:27:52.192066 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.192045 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-istio-data\") pod \"51363008-5bed-4213-9550-f106a28fcb90\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " Jul 15 12:27:52.192280 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.192081 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-credential-socket\") pod \"51363008-5bed-4213-9550-f106a28fcb90\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " Jul 15 12:27:52.192280 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.192255 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51363008-5bed-4213-9550-f106a28fcb90-istio-podinfo\") pod \"51363008-5bed-4213-9550-f106a28fcb90\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " Jul 15 12:27:52.192362 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.192310 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "51363008-5bed-4213-9550-f106a28fcb90" (UID: "51363008-5bed-4213-9550-f106a28fcb90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:27:52.192362 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.192321 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "51363008-5bed-4213-9550-f106a28fcb90" (UID: "51363008-5bed-4213-9550-f106a28fcb90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:27:52.192362 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.192332 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51363008-5bed-4213-9550-f106a28fcb90-istiod-ca-cert\") pod \"51363008-5bed-4213-9550-f106a28fcb90\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " Jul 15 12:27:52.192503 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.192399 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-istio-envoy\") pod \"51363008-5bed-4213-9550-f106a28fcb90\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " Jul 15 12:27:52.192503 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.192448 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51363008-5bed-4213-9550-f106a28fcb90-istio-token\") pod \"51363008-5bed-4213-9550-f106a28fcb90\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " Jul 15 12:27:52.192503 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.192475 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-workload-certs\") pod \"51363008-5bed-4213-9550-f106a28fcb90\" (UID: \"51363008-5bed-4213-9550-f106a28fcb90\") " Jul 15 12:27:52.192966 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.192761 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51363008-5bed-4213-9550-f106a28fcb90-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "51363008-5bed-4213-9550-f106a28fcb90" (UID: "51363008-5bed-4213-9550-f106a28fcb90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:27:52.192966 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.192954 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "51363008-5bed-4213-9550-f106a28fcb90" (UID: "51363008-5bed-4213-9550-f106a28fcb90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:27:52.193162 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.192998 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-istio-data" (OuterVolumeSpecName: "istio-data") pod "51363008-5bed-4213-9550-f106a28fcb90" (UID: "51363008-5bed-4213-9550-f106a28fcb90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:27:52.193363 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.193293 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-workload-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:27:52.193363 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.193318 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-workload-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:27:52.193363 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.193331 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-istio-data\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:27:52.193363 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.193344 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-credential-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:27:52.193363 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.193357 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51363008-5bed-4213-9550-f106a28fcb90-istiod-ca-cert\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:27:52.194660 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.194626 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51363008-5bed-4213-9550-f106a28fcb90-kube-api-access-wkd2v" (OuterVolumeSpecName: "kube-api-access-wkd2v") pod "51363008-5bed-4213-9550-f106a28fcb90" (UID: "51363008-5bed-4213-9550-f106a28fcb90"). InnerVolumeSpecName "kube-api-access-wkd2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:27:52.194793 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.194659 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51363008-5bed-4213-9550-f106a28fcb90-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "51363008-5bed-4213-9550-f106a28fcb90" (UID: "51363008-5bed-4213-9550-f106a28fcb90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:27:52.194793 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.194668 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "51363008-5bed-4213-9550-f106a28fcb90" (UID: "51363008-5bed-4213-9550-f106a28fcb90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:27:52.194926 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.194908 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51363008-5bed-4213-9550-f106a28fcb90-istio-token" (OuterVolumeSpecName: "istio-token") pod "51363008-5bed-4213-9550-f106a28fcb90" (UID: "51363008-5bed-4213-9550-f106a28fcb90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:27:52.294092 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.294062 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51363008-5bed-4213-9550-f106a28fcb90-istio-envoy\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:27:52.294092 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.294090 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51363008-5bed-4213-9550-f106a28fcb90-istio-token\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:27:52.294092 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.294100 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkd2v\" (UniqueName: \"kubernetes.io/projected/51363008-5bed-4213-9550-f106a28fcb90-kube-api-access-wkd2v\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:27:52.294294 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.294109 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51363008-5bed-4213-9550-f106a28fcb90-istio-podinfo\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:27:52.446704 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.446619 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt"] Jul 15 12:27:52.446913 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.446885 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" podUID="1b08d73a-fec3-44a7-89b5-9d53f0dd622f" containerName="maas-api" containerID="cri-o://c79e78450371b28b652deb6921869da53a7a3609a8009c5a7c8d1713c46fb870" gracePeriod=30 Jul 15 12:27:52.479528 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.479502 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:52.678363 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.678340 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" Jul 15 12:27:52.798138 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.798044 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-maas-api-tls\") pod \"1b08d73a-fec3-44a7-89b5-9d53f0dd622f\" (UID: \"1b08d73a-fec3-44a7-89b5-9d53f0dd622f\") " Jul 15 12:27:52.798138 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.798094 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-openshift-service-ca\") pod \"1b08d73a-fec3-44a7-89b5-9d53f0dd622f\" (UID: \"1b08d73a-fec3-44a7-89b5-9d53f0dd622f\") " Jul 15 12:27:52.798351 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.798240 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5f9k\" (UniqueName: \"kubernetes.io/projected/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-kube-api-access-w5f9k\") pod \"1b08d73a-fec3-44a7-89b5-9d53f0dd622f\" (UID: \"1b08d73a-fec3-44a7-89b5-9d53f0dd622f\") " Jul 15 12:27:52.798451 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.798424 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b08d73a-fec3-44a7-89b5-9d53f0dd622f" (UID: "1b08d73a-fec3-44a7-89b5-9d53f0dd622f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:27:52.800354 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.800324 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b08d73a-fec3-44a7-89b5-9d53f0dd622f" (UID: "1b08d73a-fec3-44a7-89b5-9d53f0dd622f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:27:52.800456 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.800357 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-kube-api-access-w5f9k" (OuterVolumeSpecName: "kube-api-access-w5f9k") pod "1b08d73a-fec3-44a7-89b5-9d53f0dd622f" (UID: "1b08d73a-fec3-44a7-89b5-9d53f0dd622f"). InnerVolumeSpecName "kube-api-access-w5f9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:27:52.898915 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.898884 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5f9k\" (UniqueName: \"kubernetes.io/projected/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-kube-api-access-w5f9k\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:27:52.898915 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.898913 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-maas-api-tls\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:27:52.899059 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:52.898922 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b08d73a-fec3-44a7-89b5-9d53f0dd622f-openshift-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:27:53.001801 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.001762 2570 generic.go:356] "Generic (PLEG): container finished" podID="1b08d73a-fec3-44a7-89b5-9d53f0dd622f" containerID="c79e78450371b28b652deb6921869da53a7a3609a8009c5a7c8d1713c46fb870" exitCode=0 Jul 15 12:27:53.002206 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.001816 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" Jul 15 12:27:53.002206 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.001852 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" event={"ID":"1b08d73a-fec3-44a7-89b5-9d53f0dd622f","Type":"ContainerDied","Data":"c79e78450371b28b652deb6921869da53a7a3609a8009c5a7c8d1713c46fb870"} Jul 15 12:27:53.002206 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.001904 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt" event={"ID":"1b08d73a-fec3-44a7-89b5-9d53f0dd622f","Type":"ContainerDied","Data":"44cc7772f7f49da39edb578c1702b22dcf770e0e5d7424a0fd9fda5645508a42"} Jul 15 12:27:53.002206 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.001926 2570 scope.go:117] "RemoveContainer" containerID="c79e78450371b28b652deb6921869da53a7a3609a8009c5a7c8d1713c46fb870" Jul 15 12:27:53.003445 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.003425 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" event={"ID":"51363008-5bed-4213-9550-f106a28fcb90","Type":"ContainerDied","Data":"d0b4beffabe8ddcd9d37f7dd87a3c53f9bc926a0132cfdb3a9b6b553a284934d"} Jul 15 12:27:53.003533 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.003450 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv" Jul 15 12:27:53.010777 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.010753 2570 scope.go:117] "RemoveContainer" containerID="c79e78450371b28b652deb6921869da53a7a3609a8009c5a7c8d1713c46fb870" Jul 15 12:27:53.011052 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:27:53.011033 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79e78450371b28b652deb6921869da53a7a3609a8009c5a7c8d1713c46fb870\": container with ID starting with c79e78450371b28b652deb6921869da53a7a3609a8009c5a7c8d1713c46fb870 not found: ID does not exist" containerID="c79e78450371b28b652deb6921869da53a7a3609a8009c5a7c8d1713c46fb870" Jul 15 12:27:53.011130 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.011058 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79e78450371b28b652deb6921869da53a7a3609a8009c5a7c8d1713c46fb870"} err="failed to get container status \"c79e78450371b28b652deb6921869da53a7a3609a8009c5a7c8d1713c46fb870\": rpc error: code = NotFound desc = could not find container \"c79e78450371b28b652deb6921869da53a7a3609a8009c5a7c8d1713c46fb870\": container with ID starting with c79e78450371b28b652deb6921869da53a7a3609a8009c5a7c8d1713c46fb870 not found: ID does not exist" Jul 15 12:27:53.011130 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.011073 2570 scope.go:117] "RemoveContainer" containerID="7180aa41cbebf8331bcefd574647195d82d15183369143109f4b469deb16180e" Jul 15 12:27:53.021971 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.021951 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt"] Jul 15 12:27:53.026883 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.026864 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bbac7fd2-5dfc4b6f7b-2w9zt"] Jul 15 12:27:53.035442 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.035423 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv"] Jul 15 12:27:53.039379 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.039361 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-bbac7fd2-gw-openshift-default-66d4f696dc-hb8lv"] Jul 15 12:27:53.093022 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.092957 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b08d73a-fec3-44a7-89b5-9d53f0dd622f" path="/var/lib/kubelet/pods/1b08d73a-fec3-44a7-89b5-9d53f0dd622f/volumes" Jul 15 12:27:53.093358 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.093345 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51363008-5bed-4213-9550-f106a28fcb90" path="/var/lib/kubelet/pods/51363008-5bed-4213-9550-f106a28fcb90/volumes" Jul 15 12:27:53.407333 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:53.407300 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:57.584193 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:57.584165 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:59.328090 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.328056 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:59.408806 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.408768 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:59.516705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.516672 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb"] Jul 15 12:27:59.517107 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.517087 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b08d73a-fec3-44a7-89b5-9d53f0dd622f" containerName="maas-api" Jul 15 12:27:59.517251 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.517185 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b08d73a-fec3-44a7-89b5-9d53f0dd622f" containerName="maas-api" Jul 15 12:27:59.517323 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.517210 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cfe7d47-396c-47ab-a37e-139ac9069afd" containerName="revoke-keys" Jul 15 12:27:59.517323 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.517270 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cfe7d47-396c-47ab-a37e-139ac9069afd" containerName="revoke-keys" Jul 15 12:27:59.517323 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.517297 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51363008-5bed-4213-9550-f106a28fcb90" containerName="istio-proxy" Jul 15 12:27:59.517323 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.517307 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="51363008-5bed-4213-9550-f106a28fcb90" containerName="istio-proxy" Jul 15 12:27:59.517537 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.517436 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="51363008-5bed-4213-9550-f106a28fcb90" containerName="istio-proxy" Jul 15 12:27:59.517537 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.517450 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b08d73a-fec3-44a7-89b5-9d53f0dd622f" containerName="maas-api" Jul 15 12:27:59.517537 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.517460 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cfe7d47-396c-47ab-a37e-139ac9069afd" containerName="revoke-keys" Jul 15 12:27:59.517537 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.517468 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="51363008-5bed-4213-9550-f106a28fcb90" containerName="istio-proxy" Jul 15 12:27:59.520583 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.520562 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.522722 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.522550 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6f432845-openshift-default-dockercfg-jccbk\"" Jul 15 12:27:59.522722 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.522580 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 12:27:59.531759 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.531714 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb"] Jul 15 12:27:59.557605 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.557578 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:59.662163 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.662122 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-workload-socket\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.662357 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.662183 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-envoy\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.662357 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.662233 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-data\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.662357 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.662252 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-token\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.662357 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.662281 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-podinfo\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.662357 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.662341 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-workload-certs\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.662548 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.662369 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istiod-ca-cert\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.662548 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.662388 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5sfq\" (UniqueName: \"kubernetes.io/projected/61f3ac67-8aac-45e5-b9b9-537496ed8e18-kube-api-access-x5sfq\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.662548 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.662421 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-credential-socket\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.665484 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.665447 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:27:59.763065 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.763031 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-data\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.763065 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.763066 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-token\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.763313 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.763094 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-podinfo\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.763313 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.763134 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-workload-certs\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.763313 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.763152 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istiod-ca-cert\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.763313 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.763170 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5sfq\" (UniqueName: \"kubernetes.io/projected/61f3ac67-8aac-45e5-b9b9-537496ed8e18-kube-api-access-x5sfq\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.763313 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.763198 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-credential-socket\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.763313 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.763240 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-workload-socket\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.763313 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.763289 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-envoy\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.763650 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.763498 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-data\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.763650 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.763526 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-workload-certs\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.763650 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.763564 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-credential-socket\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.763832 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.763652 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-workload-socket\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.764133 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.764067 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istiod-ca-cert\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.765604 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.765579 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-envoy\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.765716 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.765697 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-podinfo\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.773620 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.773599 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-token\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.773701 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.773640 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5sfq\" (UniqueName: \"kubernetes.io/projected/61f3ac67-8aac-45e5-b9b9-537496ed8e18-kube-api-access-x5sfq\") pod \"e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.832424 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.832390 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:27:59.968928 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.968859 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb"] Jul 15 12:27:59.971238 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:27:59.971204 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f3ac67_8aac_45e5_b9b9_537496ed8e18.slice/crio-f0fab70f673ae4de500fbcf3bf92e139dc015392dfd7ede06423da217552c924 WatchSource:0}: Error finding container f0fab70f673ae4de500fbcf3bf92e139dc015392dfd7ede06423da217552c924: Status 404 returned error can't find the container with id f0fab70f673ae4de500fbcf3bf92e139dc015392dfd7ede06423da217552c924 Jul 15 12:27:59.973318 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.973286 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:27:59.973422 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.973363 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:27:59.973422 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:27:59.973407 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:28:00.037719 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.037693 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" event={"ID":"61f3ac67-8aac-45e5-b9b9-537496ed8e18","Type":"ContainerStarted","Data":"f0fab70f673ae4de500fbcf3bf92e139dc015392dfd7ede06423da217552c924"} Jul 15 12:28:00.677150 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.677115 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:00.745078 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.745048 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s"] Jul 15 12:28:00.745413 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.745400 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51363008-5bed-4213-9550-f106a28fcb90" containerName="istio-proxy" Jul 15 12:28:00.745458 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.745415 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="51363008-5bed-4213-9550-f106a28fcb90" containerName="istio-proxy" Jul 15 12:28:00.748599 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.748577 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:28:00.750542 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.750521 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6f432845-serving-cert\"" Jul 15 12:28:00.762775 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.762750 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s"] Jul 15 12:28:00.830916 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.830882 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:00.873350 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.873310 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8nj5\" (UniqueName: \"kubernetes.io/projected/b8099f96-3501-4f07-a497-79133674036a-kube-api-access-x8nj5\") pod \"maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s\" (UID: \"b8099f96-3501-4f07-a497-79133674036a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:28:00.873525 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.873374 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8099f96-3501-4f07-a497-79133674036a-openshift-service-ca\") pod \"maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s\" (UID: \"b8099f96-3501-4f07-a497-79133674036a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:28:00.873525 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.873405 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8099f96-3501-4f07-a497-79133674036a-maas-api-tls\") pod \"maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s\" (UID: \"b8099f96-3501-4f07-a497-79133674036a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:28:00.931643 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.931569 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:00.974036 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.974003 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8nj5\" (UniqueName: \"kubernetes.io/projected/b8099f96-3501-4f07-a497-79133674036a-kube-api-access-x8nj5\") pod \"maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s\" (UID: \"b8099f96-3501-4f07-a497-79133674036a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:28:00.974167 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.974063 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8099f96-3501-4f07-a497-79133674036a-openshift-service-ca\") pod \"maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s\" (UID: \"b8099f96-3501-4f07-a497-79133674036a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:28:00.974167 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.974095 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8099f96-3501-4f07-a497-79133674036a-maas-api-tls\") pod \"maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s\" (UID: \"b8099f96-3501-4f07-a497-79133674036a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:28:00.974242 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:28:00.974230 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-serving-cert: secret "maas-api-e2e-ait-6f432845-serving-cert" not found Jul 15 12:28:00.974305 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:28:00.974295 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8099f96-3501-4f07-a497-79133674036a-maas-api-tls podName:b8099f96-3501-4f07-a497-79133674036a nodeName:}" failed. No retries permitted until 2026-07-15 12:28:01.474279 +0000 UTC m=+1748.887807329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8099f96-3501-4f07-a497-79133674036a-maas-api-tls") pod "maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" (UID: "b8099f96-3501-4f07-a497-79133674036a") : secret "maas-api-e2e-ait-6f432845-serving-cert" not found Jul 15 12:28:00.974766 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.974731 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8099f96-3501-4f07-a497-79133674036a-openshift-service-ca\") pod \"maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s\" (UID: \"b8099f96-3501-4f07-a497-79133674036a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:28:00.984704 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:00.984674 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8nj5\" (UniqueName: \"kubernetes.io/projected/b8099f96-3501-4f07-a497-79133674036a-kube-api-access-x8nj5\") pod \"maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s\" (UID: \"b8099f96-3501-4f07-a497-79133674036a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:28:01.044889 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:01.044852 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" event={"ID":"61f3ac67-8aac-45e5-b9b9-537496ed8e18","Type":"ContainerStarted","Data":"83ddfa4c9c2b6fe49d61b886d881577427c67b41c00be2d7b7acb2107f2d8344"} Jul 15 12:28:01.067105 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:01.067056 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" podStartSLOduration=2.06704191 podStartE2EDuration="2.06704191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:28:01.064447206 +0000 UTC m=+1748.477975569" watchObservedRunningTime="2026-07-15 12:28:01.06704191 +0000 UTC m=+1748.480570262" Jul 15 12:28:01.171597 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:01.171561 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:01.389859 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:01.389813 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:01.479105 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:01.479062 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8099f96-3501-4f07-a497-79133674036a-maas-api-tls\") pod \"maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s\" (UID: \"b8099f96-3501-4f07-a497-79133674036a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:28:01.481656 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:01.481627 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8099f96-3501-4f07-a497-79133674036a-maas-api-tls\") pod \"maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s\" (UID: \"b8099f96-3501-4f07-a497-79133674036a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:28:01.494085 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:01.494062 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:01.658901 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:01.658815 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:28:01.797873 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:01.797843 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s"] Jul 15 12:28:01.799941 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:28:01.799902 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8099f96_3501_4f07_a497_79133674036a.slice/crio-c214ac8c869e02c609099f643e7467f001ccaf7a373f79591b1c0f4fc2b33c72 WatchSource:0}: Error finding container c214ac8c869e02c609099f643e7467f001ccaf7a373f79591b1c0f4fc2b33c72: Status 404 returned error can't find the container with id c214ac8c869e02c609099f643e7467f001ccaf7a373f79591b1c0f4fc2b33c72 Jul 15 12:28:01.833305 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:01.833276 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:28:01.838004 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:01.837985 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:28:02.052158 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:02.052066 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" event={"ID":"b8099f96-3501-4f07-a497-79133674036a","Type":"ContainerStarted","Data":"c214ac8c869e02c609099f643e7467f001ccaf7a373f79591b1c0f4fc2b33c72"} Jul 15 12:28:02.052307 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:02.052259 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:28:02.053196 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:02.053175 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:28:02.618838 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:02.618756 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:03.059413 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:03.059314 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" event={"ID":"b8099f96-3501-4f07-a497-79133674036a","Type":"ContainerStarted","Data":"ba76fb487310849c09b4930f4eb8996db017f7a46afd13b4153c33ba2bc48a50"} Jul 15 12:28:03.085868 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:03.085821 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" podStartSLOduration=2.522701669 podStartE2EDuration="3.085805861s" totalImagesPullingTime="563.104192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:28:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:28:03.083400491 +0000 UTC m=+1750.496928843" watchObservedRunningTime="2026-07-15 12:28:03.085805861 +0000 UTC m=+1750.499334212" Jul 15 12:28:04.064501 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:04.064465 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:28:05.810190 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:05.810160 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67"] Jul 15 12:28:05.813896 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:05.813877 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" Jul 15 12:28:05.816050 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:05.816029 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tf4tp\"" Jul 15 12:28:05.821145 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:05.821045 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67"] Jul 15 12:28:05.919642 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:05.919612 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9780b98-c9c6-47a5-bb0b-7451500d2f9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f432845-xfm67\" (UID: \"a9780b98-c9c6-47a5-bb0b-7451500d2f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" Jul 15 12:28:06.020502 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:06.020464 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9780b98-c9c6-47a5-bb0b-7451500d2f9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f432845-xfm67\" (UID: \"a9780b98-c9c6-47a5-bb0b-7451500d2f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" Jul 15 12:28:06.021200 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:06.021181 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9780b98-c9c6-47a5-bb0b-7451500d2f9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f432845-xfm67\" (UID: \"a9780b98-c9c6-47a5-bb0b-7451500d2f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" Jul 15 12:28:06.125399 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:06.125368 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" Jul 15 12:28:06.301119 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:06.301094 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67"] Jul 15 12:28:06.303958 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:28:06.303924 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9780b98_c9c6_47a5_bb0b_7451500d2f9f.slice/crio-64fcadb86b8da55cd335cf53f43ca1ac5ec348880323fe5dffaf3f50329c161a WatchSource:0}: Error finding container 64fcadb86b8da55cd335cf53f43ca1ac5ec348880323fe5dffaf3f50329c161a: Status 404 returned error can't find the container with id 64fcadb86b8da55cd335cf53f43ca1ac5ec348880323fe5dffaf3f50329c161a Jul 15 12:28:07.094152 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:07.094114 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" event={"ID":"a9780b98-c9c6-47a5-bb0b-7451500d2f9f","Type":"ContainerStarted","Data":"3578934cfb374dadabb2e0dc6bb3bd0a55dd13a25805a95863995e400c76a19d"} Jul 15 12:28:07.094527 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:07.094169 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" event={"ID":"a9780b98-c9c6-47a5-bb0b-7451500d2f9f","Type":"ContainerStarted","Data":"64fcadb86b8da55cd335cf53f43ca1ac5ec348880323fe5dffaf3f50329c161a"} Jul 15 12:28:07.105518 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:07.105466 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" podStartSLOduration=2.1054472889999998 podStartE2EDuration="2.105447289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:28:07.10479877 +0000 UTC m=+1754.518327137" watchObservedRunningTime="2026-07-15 12:28:07.105447289 +0000 UTC m=+1754.518975641" Jul 15 12:28:08.095991 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:08.095956 2570 generic.go:356] "Generic (PLEG): container finished" podID="a9780b98-c9c6-47a5-bb0b-7451500d2f9f" containerID="3578934cfb374dadabb2e0dc6bb3bd0a55dd13a25805a95863995e400c76a19d" exitCode=7 Jul 15 12:28:08.096356 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:08.096041 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" event={"ID":"a9780b98-c9c6-47a5-bb0b-7451500d2f9f","Type":"ContainerDied","Data":"3578934cfb374dadabb2e0dc6bb3bd0a55dd13a25805a95863995e400c76a19d"} Jul 15 12:28:08.096356 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:08.096234 2570 scope.go:117] "RemoveContainer" containerID="3578934cfb374dadabb2e0dc6bb3bd0a55dd13a25805a95863995e400c76a19d" Jul 15 12:28:09.103453 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:09.103421 2570 generic.go:356] "Generic (PLEG): container finished" podID="a9780b98-c9c6-47a5-bb0b-7451500d2f9f" containerID="bbc5198582c7ca550601aa1b42719dcb78106720917d71c56282b0b4ba1ff876" exitCode=7 Jul 15 12:28:09.103966 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:09.103503 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" event={"ID":"a9780b98-c9c6-47a5-bb0b-7451500d2f9f","Type":"ContainerDied","Data":"bbc5198582c7ca550601aa1b42719dcb78106720917d71c56282b0b4ba1ff876"} Jul 15 12:28:09.103966 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:09.103554 2570 scope.go:117] "RemoveContainer" containerID="3578934cfb374dadabb2e0dc6bb3bd0a55dd13a25805a95863995e400c76a19d" Jul 15 12:28:09.103966 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:09.103788 2570 scope.go:117] "RemoveContainer" containerID="bbc5198582c7ca550601aa1b42719dcb78106720917d71c56282b0b4ba1ff876" Jul 15 12:28:09.104089 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:28:09.104067 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f432845-xfm67_odh-ai-gateway-infra(a9780b98-c9c6-47a5-bb0b-7451500d2f9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" podUID="a9780b98-c9c6-47a5-bb0b-7451500d2f9f" Jul 15 12:28:10.079643 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:10.079618 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:28:10.111705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:10.111679 2570 scope.go:117] "RemoveContainer" containerID="bbc5198582c7ca550601aa1b42719dcb78106720917d71c56282b0b4ba1ff876" Jul 15 12:28:10.112183 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:28:10.111973 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-6f432845-xfm67_odh-ai-gateway-infra(a9780b98-c9c6-47a5-bb0b-7451500d2f9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" podUID="a9780b98-c9c6-47a5-bb0b-7451500d2f9f" Jul 15 12:28:12.123330 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:12.123293 2570 generic.go:356] "Generic (PLEG): container finished" podID="61f3ac67-8aac-45e5-b9b9-537496ed8e18" containerID="83ddfa4c9c2b6fe49d61b886d881577427c67b41c00be2d7b7acb2107f2d8344" exitCode=0 Jul 15 12:28:12.123888 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:12.123365 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" event={"ID":"61f3ac67-8aac-45e5-b9b9-537496ed8e18","Type":"ContainerDied","Data":"83ddfa4c9c2b6fe49d61b886d881577427c67b41c00be2d7b7acb2107f2d8344"} Jul 15 12:28:12.123970 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:12.123894 2570 scope.go:117] "RemoveContainer" containerID="83ddfa4c9c2b6fe49d61b886d881577427c67b41c00be2d7b7acb2107f2d8344" Jul 15 12:28:12.124918 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:12.124879 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:28:12.125032 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:12.124957 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:28:12.125032 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:12.124991 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:28:13.130000 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:13.129965 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" event={"ID":"61f3ac67-8aac-45e5-b9b9-537496ed8e18","Type":"ContainerStarted","Data":"779eb5ee1230d35de021c89c2d3778e14c6b0d8fa6bf1415dd3179e060c30002"} Jul 15 12:28:13.833288 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:13.833249 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:28:13.834449 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:13.834422 2570 patch_prober.go:28] interesting pod/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 15 12:28:13.834563 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:13.834476 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" podUID="61f3ac67-8aac-45e5-b9b9-537496ed8e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 15 12:28:14.833406 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:14.833366 2570 patch_prober.go:28] interesting pod/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 15 12:28:14.833834 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:14.833423 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" podUID="61f3ac67-8aac-45e5-b9b9-537496ed8e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 15 12:28:15.832934 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:15.832893 2570 patch_prober.go:28] interesting pod/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 15 12:28:15.833119 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:15.832949 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" podUID="61f3ac67-8aac-45e5-b9b9-537496ed8e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 15 12:28:16.833264 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:16.833227 2570 patch_prober.go:28] interesting pod/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 15 12:28:16.833660 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:16.833282 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" podUID="61f3ac67-8aac-45e5-b9b9-537496ed8e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 15 12:28:17.052690 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:17.052650 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:28:17.837070 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:17.837038 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:28:17.838043 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:17.838025 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:28:25.086701 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:25.086668 2570 scope.go:117] "RemoveContainer" containerID="bbc5198582c7ca550601aa1b42719dcb78106720917d71c56282b0b4ba1ff876" Jul 15 12:28:26.203225 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:26.203194 2570 generic.go:356] "Generic (PLEG): container finished" podID="a9780b98-c9c6-47a5-bb0b-7451500d2f9f" containerID="96830b73b70abc9c9720a50f579c2be6e78940778889a44b224b5ea470e0fc75" exitCode=0 Jul 15 12:28:26.203646 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:26.203230 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" event={"ID":"a9780b98-c9c6-47a5-bb0b-7451500d2f9f","Type":"ContainerDied","Data":"96830b73b70abc9c9720a50f579c2be6e78940778889a44b224b5ea470e0fc75"} Jul 15 12:28:26.203646 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:26.203279 2570 scope.go:117] "RemoveContainer" containerID="bbc5198582c7ca550601aa1b42719dcb78106720917d71c56282b0b4ba1ff876" Jul 15 12:28:27.236480 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:27.236447 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67"] Jul 15 12:28:27.346538 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:27.346513 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" Jul 15 12:28:27.419180 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:27.419152 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9780b98-c9c6-47a5-bb0b-7451500d2f9f-maas-api-service-ca\") pod \"a9780b98-c9c6-47a5-bb0b-7451500d2f9f\" (UID: \"a9780b98-c9c6-47a5-bb0b-7451500d2f9f\") " Jul 15 12:28:27.419491 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:27.419471 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9780b98-c9c6-47a5-bb0b-7451500d2f9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9780b98-c9c6-47a5-bb0b-7451500d2f9f" (UID: "a9780b98-c9c6-47a5-bb0b-7451500d2f9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:28:27.519978 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:27.519882 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9780b98-c9c6-47a5-bb0b-7451500d2f9f-maas-api-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:28:28.215459 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:28.215423 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" event={"ID":"a9780b98-c9c6-47a5-bb0b-7451500d2f9f","Type":"ContainerDied","Data":"64fcadb86b8da55cd335cf53f43ca1ac5ec348880323fe5dffaf3f50329c161a"} Jul 15 12:28:28.215604 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:28.215501 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64fcadb86b8da55cd335cf53f43ca1ac5ec348880323fe5dffaf3f50329c161a" Jul 15 12:28:28.215604 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:28.215434 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67" Jul 15 12:28:28.242133 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:28.242104 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67"] Jul 15 12:28:28.247841 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:28.247814 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-xfm67"] Jul 15 12:28:29.091829 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:29.091794 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9780b98-c9c6-47a5-bb0b-7451500d2f9f" path="/var/lib/kubelet/pods/a9780b98-c9c6-47a5-bb0b-7451500d2f9f/volumes" Jul 15 12:28:49.398906 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:49.398825 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:49.499796 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:49.499714 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:53.216933 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:53.216906 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:28:53.222651 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:28:53.222629 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:29:05.862826 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:05.862782 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-7td4p"] Jul 15 12:29:05.863307 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:05.863177 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9780b98-c9c6-47a5-bb0b-7451500d2f9f" containerName="revoke-keys" Jul 15 12:29:05.863307 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:05.863189 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9780b98-c9c6-47a5-bb0b-7451500d2f9f" containerName="revoke-keys" Jul 15 12:29:05.863307 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:05.863213 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9780b98-c9c6-47a5-bb0b-7451500d2f9f" containerName="revoke-keys" Jul 15 12:29:05.863307 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:05.863220 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9780b98-c9c6-47a5-bb0b-7451500d2f9f" containerName="revoke-keys" Jul 15 12:29:05.863307 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:05.863290 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9780b98-c9c6-47a5-bb0b-7451500d2f9f" containerName="revoke-keys" Jul 15 12:29:05.863307 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:05.863301 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9780b98-c9c6-47a5-bb0b-7451500d2f9f" containerName="revoke-keys" Jul 15 12:29:05.865517 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:05.865496 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-7td4p" Jul 15 12:29:05.867408 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:05.867389 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tf4tp\"" Jul 15 12:29:05.876172 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:05.876146 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-7td4p"] Jul 15 12:29:05.971880 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:05.971849 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b487df91-72d9-4526-b4c3-965aee774a8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f432845-7td4p\" (UID: \"b487df91-72d9-4526-b4c3-965aee774a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-7td4p" Jul 15 12:29:06.072783 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:06.072724 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b487df91-72d9-4526-b4c3-965aee774a8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f432845-7td4p\" (UID: \"b487df91-72d9-4526-b4c3-965aee774a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-7td4p" Jul 15 12:29:06.073314 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:06.073292 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b487df91-72d9-4526-b4c3-965aee774a8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6f432845-7td4p\" (UID: \"b487df91-72d9-4526-b4c3-965aee774a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-7td4p" Jul 15 12:29:06.176013 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:06.175916 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-7td4p" Jul 15 12:29:06.312750 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:06.312710 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-7td4p"] Jul 15 12:29:06.315829 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:29:06.315730 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb487df91_72d9_4526_b4c3_965aee774a8c.slice/crio-8d7a456a030ece4fe32b979aa9c58d81e744d35af2371c8ff628973abff6bec4 WatchSource:0}: Error finding container 8d7a456a030ece4fe32b979aa9c58d81e744d35af2371c8ff628973abff6bec4: Status 404 returned error can't find the container with id 8d7a456a030ece4fe32b979aa9c58d81e744d35af2371c8ff628973abff6bec4 Jul 15 12:29:06.398294 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:06.398255 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:06.402211 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:06.402179 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-7td4p" event={"ID":"b487df91-72d9-4526-b4c3-965aee774a8c","Type":"ContainerStarted","Data":"8d7a456a030ece4fe32b979aa9c58d81e744d35af2371c8ff628973abff6bec4"} Jul 15 12:29:06.499249 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:06.499219 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:07.408795 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:07.408764 2570 generic.go:356] "Generic (PLEG): container finished" podID="b487df91-72d9-4526-b4c3-965aee774a8c" containerID="cf416f185b5d1ee9e59b20c5523b0a7a3695646aadc36e769221f5a6b9e3452e" exitCode=0 Jul 15 12:29:07.409184 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:07.408804 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-7td4p" event={"ID":"b487df91-72d9-4526-b4c3-965aee774a8c","Type":"ContainerDied","Data":"cf416f185b5d1ee9e59b20c5523b0a7a3695646aadc36e769221f5a6b9e3452e"} Jul 15 12:29:08.544230 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:08.544196 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-7td4p" Jul 15 12:29:08.597904 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:08.597870 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b487df91-72d9-4526-b4c3-965aee774a8c-maas-api-service-ca\") pod \"b487df91-72d9-4526-b4c3-965aee774a8c\" (UID: \"b487df91-72d9-4526-b4c3-965aee774a8c\") " Jul 15 12:29:08.598238 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:08.598217 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b487df91-72d9-4526-b4c3-965aee774a8c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b487df91-72d9-4526-b4c3-965aee774a8c" (UID: "b487df91-72d9-4526-b4c3-965aee774a8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:29:08.698927 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:08.698843 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b487df91-72d9-4526-b4c3-965aee774a8c-maas-api-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:29:09.421505 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:09.421425 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-7td4p" Jul 15 12:29:09.421505 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:09.421464 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6f432845-7td4p" event={"ID":"b487df91-72d9-4526-b4c3-965aee774a8c","Type":"ContainerDied","Data":"8d7a456a030ece4fe32b979aa9c58d81e744d35af2371c8ff628973abff6bec4"} Jul 15 12:29:09.421505 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:09.421506 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d7a456a030ece4fe32b979aa9c58d81e744d35af2371c8ff628973abff6bec4" Jul 15 12:29:11.041871 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:11.041838 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:11.142551 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:11.142513 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:16.450678 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:16.450634 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s"] Jul 15 12:29:16.451111 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:16.450894 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" podUID="b8099f96-3501-4f07-a497-79133674036a" containerName="maas-api" containerID="cri-o://ba76fb487310849c09b4930f4eb8996db017f7a46afd13b4153c33ba2bc48a50" gracePeriod=30 Jul 15 12:29:16.489525 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:16.489492 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:16.704061 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:16.703995 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:29:16.770586 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:16.770550 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8099f96-3501-4f07-a497-79133674036a-openshift-service-ca\") pod \"b8099f96-3501-4f07-a497-79133674036a\" (UID: \"b8099f96-3501-4f07-a497-79133674036a\") " Jul 15 12:29:16.770781 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:16.770622 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8nj5\" (UniqueName: \"kubernetes.io/projected/b8099f96-3501-4f07-a497-79133674036a-kube-api-access-x8nj5\") pod \"b8099f96-3501-4f07-a497-79133674036a\" (UID: \"b8099f96-3501-4f07-a497-79133674036a\") " Jul 15 12:29:16.770781 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:16.770654 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8099f96-3501-4f07-a497-79133674036a-maas-api-tls\") pod \"b8099f96-3501-4f07-a497-79133674036a\" (UID: \"b8099f96-3501-4f07-a497-79133674036a\") " Jul 15 12:29:16.770968 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:16.770945 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8099f96-3501-4f07-a497-79133674036a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b8099f96-3501-4f07-a497-79133674036a" (UID: "b8099f96-3501-4f07-a497-79133674036a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:29:16.772885 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:16.772853 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8099f96-3501-4f07-a497-79133674036a-kube-api-access-x8nj5" (OuterVolumeSpecName: "kube-api-access-x8nj5") pod "b8099f96-3501-4f07-a497-79133674036a" (UID: "b8099f96-3501-4f07-a497-79133674036a"). InnerVolumeSpecName "kube-api-access-x8nj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:29:16.772998 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:16.772886 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8099f96-3501-4f07-a497-79133674036a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b8099f96-3501-4f07-a497-79133674036a" (UID: "b8099f96-3501-4f07-a497-79133674036a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:29:16.871295 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:16.871260 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8nj5\" (UniqueName: \"kubernetes.io/projected/b8099f96-3501-4f07-a497-79133674036a-kube-api-access-x8nj5\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:29:16.871295 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:16.871287 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8099f96-3501-4f07-a497-79133674036a-maas-api-tls\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:29:16.871295 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:16.871298 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8099f96-3501-4f07-a497-79133674036a-openshift-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:29:17.467777 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:17.467665 2570 generic.go:356] "Generic (PLEG): container finished" podID="b8099f96-3501-4f07-a497-79133674036a" containerID="ba76fb487310849c09b4930f4eb8996db017f7a46afd13b4153c33ba2bc48a50" exitCode=0 Jul 15 12:29:17.467777 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:17.467709 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" Jul 15 12:29:17.467777 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:17.467763 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" event={"ID":"b8099f96-3501-4f07-a497-79133674036a","Type":"ContainerDied","Data":"ba76fb487310849c09b4930f4eb8996db017f7a46afd13b4153c33ba2bc48a50"} Jul 15 12:29:17.468338 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:17.467818 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s" event={"ID":"b8099f96-3501-4f07-a497-79133674036a","Type":"ContainerDied","Data":"c214ac8c869e02c609099f643e7467f001ccaf7a373f79591b1c0f4fc2b33c72"} Jul 15 12:29:17.468338 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:17.467841 2570 scope.go:117] "RemoveContainer" containerID="ba76fb487310849c09b4930f4eb8996db017f7a46afd13b4153c33ba2bc48a50" Jul 15 12:29:17.476696 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:17.476673 2570 scope.go:117] "RemoveContainer" containerID="ba76fb487310849c09b4930f4eb8996db017f7a46afd13b4153c33ba2bc48a50" Jul 15 12:29:17.476948 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:29:17.476931 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba76fb487310849c09b4930f4eb8996db017f7a46afd13b4153c33ba2bc48a50\": container with ID starting with ba76fb487310849c09b4930f4eb8996db017f7a46afd13b4153c33ba2bc48a50 not found: ID does not exist" containerID="ba76fb487310849c09b4930f4eb8996db017f7a46afd13b4153c33ba2bc48a50" Jul 15 12:29:17.477022 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:17.476958 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba76fb487310849c09b4930f4eb8996db017f7a46afd13b4153c33ba2bc48a50"} err="failed to get container status \"ba76fb487310849c09b4930f4eb8996db017f7a46afd13b4153c33ba2bc48a50\": rpc error: code = NotFound desc = could not find container \"ba76fb487310849c09b4930f4eb8996db017f7a46afd13b4153c33ba2bc48a50\": container with ID starting with ba76fb487310849c09b4930f4eb8996db017f7a46afd13b4153c33ba2bc48a50 not found: ID does not exist" Jul 15 12:29:17.483840 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:17.483813 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s"] Jul 15 12:29:17.486531 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:17.486506 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6f432845-7df74d5cb8-vcf7s"] Jul 15 12:29:19.091928 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:19.091841 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8099f96-3501-4f07-a497-79133674036a" path="/var/lib/kubelet/pods/b8099f96-3501-4f07-a497-79133674036a/volumes" Jul 15 12:29:20.958029 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:20.957993 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:27.027244 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:27.027209 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:27.126105 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:27.126071 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:31.943910 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:31.943874 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb"] Jul 15 12:29:31.944339 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:31.944071 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" podUID="61f3ac67-8aac-45e5-b9b9-537496ed8e18" containerName="istio-proxy" containerID="cri-o://779eb5ee1230d35de021c89c2d3778e14c6b0d8fa6bf1415dd3179e060c30002" gracePeriod=30 Jul 15 12:29:31.975287 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:31.975243 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:32.241638 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.241553 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp"] Jul 15 12:29:32.242204 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.242186 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b487df91-72d9-4526-b4c3-965aee774a8c" containerName="revoke-keys" Jul 15 12:29:32.242262 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.242209 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b487df91-72d9-4526-b4c3-965aee774a8c" containerName="revoke-keys" Jul 15 12:29:32.242262 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.242232 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9780b98-c9c6-47a5-bb0b-7451500d2f9f" containerName="revoke-keys" Jul 15 12:29:32.242262 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.242242 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9780b98-c9c6-47a5-bb0b-7451500d2f9f" containerName="revoke-keys" Jul 15 12:29:32.242262 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.242257 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8099f96-3501-4f07-a497-79133674036a" containerName="maas-api" Jul 15 12:29:32.242396 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.242266 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8099f96-3501-4f07-a497-79133674036a" containerName="maas-api" Jul 15 12:29:32.242396 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.242351 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9780b98-c9c6-47a5-bb0b-7451500d2f9f" containerName="revoke-keys" Jul 15 12:29:32.242396 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.242368 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8099f96-3501-4f07-a497-79133674036a" containerName="maas-api" Jul 15 12:29:32.242396 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.242378 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b487df91-72d9-4526-b4c3-965aee774a8c" containerName="revoke-keys" Jul 15 12:29:32.245838 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.245812 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.251800 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.247907 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-99db84ea-openshift-default-dockercfg-k58p4\"" Jul 15 12:29:32.257877 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.257843 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp"] Jul 15 12:29:32.278572 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.278534 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:32.316647 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.316610 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-credential-socket\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.316647 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.316652 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e867d7c-fd95-4283-858c-4e843a136781-istio-podinfo\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.316963 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.316673 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-istio-data\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.316963 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.316774 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-workload-socket\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.316963 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.316823 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e867d7c-fd95-4283-858c-4e843a136781-istio-token\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.316963 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.316869 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e867d7c-fd95-4283-858c-4e843a136781-istiod-ca-cert\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.316963 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.316900 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-workload-certs\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.317175 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.316969 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxj6b\" (UniqueName: \"kubernetes.io/projected/7e867d7c-fd95-4283-858c-4e843a136781-kube-api-access-pxj6b\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.317175 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.317005 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-istio-envoy\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.418458 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.418414 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxj6b\" (UniqueName: \"kubernetes.io/projected/7e867d7c-fd95-4283-858c-4e843a136781-kube-api-access-pxj6b\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.418716 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.418466 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-istio-envoy\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.418716 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.418518 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-credential-socket\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.418716 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.418543 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e867d7c-fd95-4283-858c-4e843a136781-istio-podinfo\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.418716 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.418689 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-istio-data\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.419007 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.418824 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-workload-socket\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.419007 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.418852 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e867d7c-fd95-4283-858c-4e843a136781-istio-token\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.419007 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.418881 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e867d7c-fd95-4283-858c-4e843a136781-istiod-ca-cert\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.419007 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.418910 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-workload-certs\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.419007 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.418981 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-credential-socket\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.419305 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.419062 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-istio-data\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.419305 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.419165 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-workload-socket\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.419305 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.419263 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-workload-certs\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.419630 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.419523 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e867d7c-fd95-4283-858c-4e843a136781-istiod-ca-cert\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.421143 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.421121 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-istio-envoy\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.421228 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.421159 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e867d7c-fd95-4283-858c-4e843a136781-istio-podinfo\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.426292 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.426269 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e867d7c-fd95-4283-858c-4e843a136781-istio-token\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.426490 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.426462 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxj6b\" (UniqueName: \"kubernetes.io/projected/7e867d7c-fd95-4283-858c-4e843a136781-kube-api-access-pxj6b\") pod \"e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.560341 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.560248 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:32.704166 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.704136 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp"] Jul 15 12:29:32.706920 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:29:32.706895 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e867d7c_fd95_4283_858c_4e843a136781.slice/crio-18ebc261d2ae16e0eb4fab8c26b8c97051288476b01cd1fb06964cd1b2cbceb1 WatchSource:0}: Error finding container 18ebc261d2ae16e0eb4fab8c26b8c97051288476b01cd1fb06964cd1b2cbceb1: Status 404 returned error can't find the container with id 18ebc261d2ae16e0eb4fab8c26b8c97051288476b01cd1fb06964cd1b2cbceb1 Jul 15 12:29:32.709613 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.709569 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:29:32.709685 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.709665 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:29:32.709728 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.709713 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:29:32.839405 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.839318 2570 patch_prober.go:28] interesting pod/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 12:29:32.839405 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:32.839390 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" podUID="61f3ac67-8aac-45e5-b9b9-537496ed8e18" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 12:29:33.415304 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.415243 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:33.500262 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.500223 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc"] Jul 15 12:29:33.504144 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.504123 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:29:33.506060 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.506039 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-99db84ea-serving-cert\"" Jul 15 12:29:33.514911 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.514883 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc"] Jul 15 12:29:33.520087 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.520059 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:33.545476 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.545434 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" event={"ID":"7e867d7c-fd95-4283-858c-4e843a136781","Type":"ContainerStarted","Data":"262ef56128a15e934d127a0720bb2083d6f1aaabc6cfebbc772682bb8c729d7f"} Jul 15 12:29:33.545666 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.545499 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" event={"ID":"7e867d7c-fd95-4283-858c-4e843a136781","Type":"ContainerStarted","Data":"18ebc261d2ae16e0eb4fab8c26b8c97051288476b01cd1fb06964cd1b2cbceb1"} Jul 15 12:29:33.560670 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.560640 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:33.565588 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.565552 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:33.567771 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.567710 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" podStartSLOduration=1.5676958189999999 podStartE2EDuration="1.567695819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:29:33.565625679 +0000 UTC m=+1840.979154034" watchObservedRunningTime="2026-07-15 12:29:33.567695819 +0000 UTC m=+1840.981224170" Jul 15 12:29:33.605694 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.605639 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:33.631994 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.631953 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8db1effd-21ff-4206-a038-922a6318afc4-maas-api-tls\") pod \"maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc\" (UID: \"8db1effd-21ff-4206-a038-922a6318afc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:29:33.632204 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.632016 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db1effd-21ff-4206-a038-922a6318afc4-openshift-service-ca\") pod \"maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc\" (UID: \"8db1effd-21ff-4206-a038-922a6318afc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:29:33.632204 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.632052 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxkcw\" (UniqueName: \"kubernetes.io/projected/8db1effd-21ff-4206-a038-922a6318afc4-kube-api-access-zxkcw\") pod \"maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc\" (UID: \"8db1effd-21ff-4206-a038-922a6318afc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:29:33.732796 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.732695 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8db1effd-21ff-4206-a038-922a6318afc4-maas-api-tls\") pod \"maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc\" (UID: \"8db1effd-21ff-4206-a038-922a6318afc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:29:33.732796 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.732786 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db1effd-21ff-4206-a038-922a6318afc4-openshift-service-ca\") pod \"maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc\" (UID: \"8db1effd-21ff-4206-a038-922a6318afc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:29:33.733033 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.732814 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxkcw\" (UniqueName: \"kubernetes.io/projected/8db1effd-21ff-4206-a038-922a6318afc4-kube-api-access-zxkcw\") pod \"maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc\" (UID: \"8db1effd-21ff-4206-a038-922a6318afc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:29:33.733033 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:29:33.732862 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-serving-cert: secret "maas-api-e2e-derive-99db84ea-serving-cert" not found Jul 15 12:29:33.733033 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:29:33.732921 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8db1effd-21ff-4206-a038-922a6318afc4-maas-api-tls podName:8db1effd-21ff-4206-a038-922a6318afc4 nodeName:}" failed. No retries permitted until 2026-07-15 12:29:34.232904931 +0000 UTC m=+1841.646433264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8db1effd-21ff-4206-a038-922a6318afc4-maas-api-tls") pod "maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" (UID: "8db1effd-21ff-4206-a038-922a6318afc4") : secret "maas-api-e2e-derive-99db84ea-serving-cert" not found Jul 15 12:29:33.733479 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.733454 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db1effd-21ff-4206-a038-922a6318afc4-openshift-service-ca\") pod \"maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc\" (UID: \"8db1effd-21ff-4206-a038-922a6318afc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:29:33.743782 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.743759 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxkcw\" (UniqueName: \"kubernetes.io/projected/8db1effd-21ff-4206-a038-922a6318afc4-kube-api-access-zxkcw\") pod \"maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc\" (UID: \"8db1effd-21ff-4206-a038-922a6318afc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:29:33.815219 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:33.815176 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:34.237856 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:34.237819 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8db1effd-21ff-4206-a038-922a6318afc4-maas-api-tls\") pod \"maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc\" (UID: \"8db1effd-21ff-4206-a038-922a6318afc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:29:34.240477 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:34.240453 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8db1effd-21ff-4206-a038-922a6318afc4-maas-api-tls\") pod \"maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc\" (UID: \"8db1effd-21ff-4206-a038-922a6318afc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:29:34.362310 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:34.362202 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:34.416608 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:34.416572 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:29:34.462705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:34.462674 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:34.552531 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:34.552501 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:34.553515 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:34.553496 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:34.567675 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:34.567648 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc"] Jul 15 12:29:34.570936 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:29:34.570908 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db1effd_21ff_4206_a038_922a6318afc4.slice/crio-57c64e42435f65ec6fcb237f341fa29c57261bce1d1432711a654165be15864b WatchSource:0}: Error finding container 57c64e42435f65ec6fcb237f341fa29c57261bce1d1432711a654165be15864b: Status 404 returned error can't find the container with id 57c64e42435f65ec6fcb237f341fa29c57261bce1d1432711a654165be15864b Jul 15 12:29:35.557974 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:35.557941 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" event={"ID":"8db1effd-21ff-4206-a038-922a6318afc4","Type":"ContainerStarted","Data":"57c64e42435f65ec6fcb237f341fa29c57261bce1d1432711a654165be15864b"} Jul 15 12:29:37.240761 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.240723 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:29:37.386766 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.386058 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-credential-socket\") pod \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " Jul 15 12:29:37.386766 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.386135 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-envoy\") pod \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " Jul 15 12:29:37.386766 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.386164 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5sfq\" (UniqueName: \"kubernetes.io/projected/61f3ac67-8aac-45e5-b9b9-537496ed8e18-kube-api-access-x5sfq\") pod \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " Jul 15 12:29:37.386766 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.386221 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-workload-certs\") pod \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " Jul 15 12:29:37.386766 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.386263 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istiod-ca-cert\") pod \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " Jul 15 12:29:37.386766 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.386307 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-workload-socket\") pod \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " Jul 15 12:29:37.386766 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.386389 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-token\") pod \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " Jul 15 12:29:37.386766 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.386420 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-data\") pod \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " Jul 15 12:29:37.386766 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.386469 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-podinfo\") pod \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\" (UID: \"61f3ac67-8aac-45e5-b9b9-537496ed8e18\") " Jul 15 12:29:37.387382 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.387071 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "61f3ac67-8aac-45e5-b9b9-537496ed8e18" (UID: "61f3ac67-8aac-45e5-b9b9-537496ed8e18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:29:37.387382 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.387079 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "61f3ac67-8aac-45e5-b9b9-537496ed8e18" (UID: "61f3ac67-8aac-45e5-b9b9-537496ed8e18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:29:37.387382 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.387221 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "61f3ac67-8aac-45e5-b9b9-537496ed8e18" (UID: "61f3ac67-8aac-45e5-b9b9-537496ed8e18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:29:37.387382 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.387271 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "61f3ac67-8aac-45e5-b9b9-537496ed8e18" (UID: "61f3ac67-8aac-45e5-b9b9-537496ed8e18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:29:37.388479 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.388442 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-data" (OuterVolumeSpecName: "istio-data") pod "61f3ac67-8aac-45e5-b9b9-537496ed8e18" (UID: "61f3ac67-8aac-45e5-b9b9-537496ed8e18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:29:37.389411 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.389387 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "61f3ac67-8aac-45e5-b9b9-537496ed8e18" (UID: "61f3ac67-8aac-45e5-b9b9-537496ed8e18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:29:37.389548 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.389453 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-token" (OuterVolumeSpecName: "istio-token") pod "61f3ac67-8aac-45e5-b9b9-537496ed8e18" (UID: "61f3ac67-8aac-45e5-b9b9-537496ed8e18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:29:37.389823 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.389793 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f3ac67-8aac-45e5-b9b9-537496ed8e18-kube-api-access-x5sfq" (OuterVolumeSpecName: "kube-api-access-x5sfq") pod "61f3ac67-8aac-45e5-b9b9-537496ed8e18" (UID: "61f3ac67-8aac-45e5-b9b9-537496ed8e18"). InnerVolumeSpecName "kube-api-access-x5sfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:29:37.390852 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.390826 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "61f3ac67-8aac-45e5-b9b9-537496ed8e18" (UID: "61f3ac67-8aac-45e5-b9b9-537496ed8e18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:29:37.487420 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.487326 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-envoy\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:29:37.487420 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.487362 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5sfq\" (UniqueName: \"kubernetes.io/projected/61f3ac67-8aac-45e5-b9b9-537496ed8e18-kube-api-access-x5sfq\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:29:37.487420 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.487373 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-workload-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:29:37.487420 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.487382 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istiod-ca-cert\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:29:37.487420 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.487390 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-workload-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:29:37.487420 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.487398 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-token\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:29:37.487420 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.487405 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-data\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:29:37.487420 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.487413 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f3ac67-8aac-45e5-b9b9-537496ed8e18-istio-podinfo\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:29:37.487420 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.487420 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f3ac67-8aac-45e5-b9b9-537496ed8e18-credential-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:29:37.572871 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.572829 2570 generic.go:356] "Generic (PLEG): container finished" podID="61f3ac67-8aac-45e5-b9b9-537496ed8e18" containerID="779eb5ee1230d35de021c89c2d3778e14c6b0d8fa6bf1415dd3179e060c30002" exitCode=0 Jul 15 12:29:37.573068 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.572976 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" event={"ID":"61f3ac67-8aac-45e5-b9b9-537496ed8e18","Type":"ContainerDied","Data":"779eb5ee1230d35de021c89c2d3778e14c6b0d8fa6bf1415dd3179e060c30002"} Jul 15 12:29:37.573068 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.573063 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" event={"ID":"61f3ac67-8aac-45e5-b9b9-537496ed8e18","Type":"ContainerDied","Data":"f0fab70f673ae4de500fbcf3bf92e139dc015392dfd7ede06423da217552c924"} Jul 15 12:29:37.573189 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.573095 2570 scope.go:117] "RemoveContainer" containerID="779eb5ee1230d35de021c89c2d3778e14c6b0d8fa6bf1415dd3179e060c30002" Jul 15 12:29:37.573298 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.573279 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb" Jul 15 12:29:37.597187 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.597166 2570 scope.go:117] "RemoveContainer" containerID="83ddfa4c9c2b6fe49d61b886d881577427c67b41c00be2d7b7acb2107f2d8344" Jul 15 12:29:37.605495 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.605468 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb"] Jul 15 12:29:37.605849 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.605831 2570 scope.go:117] "RemoveContainer" containerID="779eb5ee1230d35de021c89c2d3778e14c6b0d8fa6bf1415dd3179e060c30002" Jul 15 12:29:37.606101 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:29:37.606082 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779eb5ee1230d35de021c89c2d3778e14c6b0d8fa6bf1415dd3179e060c30002\": container with ID starting with 779eb5ee1230d35de021c89c2d3778e14c6b0d8fa6bf1415dd3179e060c30002 not found: ID does not exist" containerID="779eb5ee1230d35de021c89c2d3778e14c6b0d8fa6bf1415dd3179e060c30002" Jul 15 12:29:37.606154 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.606110 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779eb5ee1230d35de021c89c2d3778e14c6b0d8fa6bf1415dd3179e060c30002"} err="failed to get container status \"779eb5ee1230d35de021c89c2d3778e14c6b0d8fa6bf1415dd3179e060c30002\": rpc error: code = NotFound desc = could not find container \"779eb5ee1230d35de021c89c2d3778e14c6b0d8fa6bf1415dd3179e060c30002\": container with ID starting with 779eb5ee1230d35de021c89c2d3778e14c6b0d8fa6bf1415dd3179e060c30002 not found: ID does not exist" Jul 15 12:29:37.606154 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.606130 2570 scope.go:117] "RemoveContainer" containerID="83ddfa4c9c2b6fe49d61b886d881577427c67b41c00be2d7b7acb2107f2d8344" Jul 15 12:29:37.606318 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:29:37.606301 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ddfa4c9c2b6fe49d61b886d881577427c67b41c00be2d7b7acb2107f2d8344\": container with ID starting with 83ddfa4c9c2b6fe49d61b886d881577427c67b41c00be2d7b7acb2107f2d8344 not found: ID does not exist" containerID="83ddfa4c9c2b6fe49d61b886d881577427c67b41c00be2d7b7acb2107f2d8344" Jul 15 12:29:37.606357 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.606324 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ddfa4c9c2b6fe49d61b886d881577427c67b41c00be2d7b7acb2107f2d8344"} err="failed to get container status \"83ddfa4c9c2b6fe49d61b886d881577427c67b41c00be2d7b7acb2107f2d8344\": rpc error: code = NotFound desc = could not find container \"83ddfa4c9c2b6fe49d61b886d881577427c67b41c00be2d7b7acb2107f2d8344\": container with ID starting with 83ddfa4c9c2b6fe49d61b886d881577427c67b41c00be2d7b7acb2107f2d8344 not found: ID does not exist" Jul 15 12:29:37.611817 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:37.611797 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6f432845-openshift-default-764b7dc9c5-kdmbb"] Jul 15 12:29:38.582571 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:38.582531 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" event={"ID":"8db1effd-21ff-4206-a038-922a6318afc4","Type":"ContainerStarted","Data":"2ba5e624aca7fe4b3de64131c84b057d7f57df5dbbfa114830b1126388656091"} Jul 15 12:29:38.583970 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:38.583945 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:29:38.604760 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:38.604682 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" podStartSLOduration=2.128044328 podStartE2EDuration="5.604659881s" totalImagesPullingTime="3.476615553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:29:38.602653807 +0000 UTC m=+1846.016182191" watchObservedRunningTime="2026-07-15 12:29:38.604659881 +0000 UTC m=+1846.018188234" Jul 15 12:29:38.680426 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:38.680387 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:39.092941 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:39.092907 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61f3ac67-8aac-45e5-b9b9-537496ed8e18" path="/var/lib/kubelet/pods/61f3ac67-8aac-45e5-b9b9-537496ed8e18/volumes" Jul 15 12:29:41.326724 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:41.326614 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:44.616948 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:44.616909 2570 generic.go:356] "Generic (PLEG): container finished" podID="7e867d7c-fd95-4283-858c-4e843a136781" containerID="262ef56128a15e934d127a0720bb2083d6f1aaabc6cfebbc772682bb8c729d7f" exitCode=0 Jul 15 12:29:44.617368 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:44.616987 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" event={"ID":"7e867d7c-fd95-4283-858c-4e843a136781","Type":"ContainerDied","Data":"262ef56128a15e934d127a0720bb2083d6f1aaabc6cfebbc772682bb8c729d7f"} Jul 15 12:29:44.617414 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:44.617367 2570 scope.go:117] "RemoveContainer" containerID="262ef56128a15e934d127a0720bb2083d6f1aaabc6cfebbc772682bb8c729d7f" Jul 15 12:29:44.618511 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:44.618477 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:29:44.618584 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:44.618566 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:29:44.618636 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:44.618620 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:29:44.740788 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:44.740757 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:45.604501 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:45.604457 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:29:45.624469 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:45.624430 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" event={"ID":"7e867d7c-fd95-4283-858c-4e843a136781","Type":"ContainerStarted","Data":"51673f9fc48a14c574aba40c0fea13ce0f1e35a9269372df49f3c2e22be41e20"} Jul 15 12:29:46.560983 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:46.560940 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:46.562341 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:46.562299 2570 patch_prober.go:28] interesting pod/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 15 12:29:46.562505 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:46.562376 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" podUID="7e867d7c-fd95-4283-858c-4e843a136781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 15 12:29:47.561128 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:47.561083 2570 patch_prober.go:28] interesting pod/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 15 12:29:47.561516 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:47.561149 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" podUID="7e867d7c-fd95-4283-858c-4e843a136781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 15 12:29:48.561633 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:48.561595 2570 patch_prober.go:28] interesting pod/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 15 12:29:48.562167 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:48.561654 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" podUID="7e867d7c-fd95-4283-858c-4e843a136781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 15 12:29:49.552673 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:49.552631 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:49.561458 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:49.561430 2570 patch_prober.go:28] interesting pod/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 15 12:29:49.561536 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:49.561512 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" podUID="7e867d7c-fd95-4283-858c-4e843a136781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 15 12:29:50.565237 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:50.565203 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:29:50.566213 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:29:50.566193 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:30:00.132943 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.132905 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735310-r9kxg"] Jul 15 12:30:00.133428 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.133303 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61f3ac67-8aac-45e5-b9b9-537496ed8e18" containerName="istio-proxy" Jul 15 12:30:00.133428 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.133315 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="61f3ac67-8aac-45e5-b9b9-537496ed8e18" containerName="istio-proxy" Jul 15 12:30:00.133428 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.133326 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61f3ac67-8aac-45e5-b9b9-537496ed8e18" containerName="istio-proxy" Jul 15 12:30:00.133428 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.133332 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="61f3ac67-8aac-45e5-b9b9-537496ed8e18" containerName="istio-proxy" Jul 15 12:30:00.133428 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.133394 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="61f3ac67-8aac-45e5-b9b9-537496ed8e18" containerName="istio-proxy" Jul 15 12:30:00.133428 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.133402 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="61f3ac67-8aac-45e5-b9b9-537496ed8e18" containerName="istio-proxy" Jul 15 12:30:00.136278 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.136262 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-99db84ea-29735310-zgk48"] Jul 15 12:30:00.136405 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.136388 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735310-r9kxg" Jul 15 12:30:00.138831 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.138805 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tf4tp\"" Jul 15 12:30:00.139579 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.139550 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-99db84ea-29735310-zgk48" Jul 15 12:30:00.145952 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.145873 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-99db84ea-29735310-zgk48"] Jul 15 12:30:00.150587 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.150557 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735310-r9kxg"] Jul 15 12:30:00.158048 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.158020 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735310-r9kxg" Jul 15 12:30:00.166016 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.165990 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-99db84ea-29735310-zgk48" Jul 15 12:30:00.311632 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.311606 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735310-r9kxg"] Jul 15 12:30:00.314428 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:30:00.314396 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f0d292_b419_4b2f_81de_7c25e974a297.slice/crio-a11310fc561143c1941fda0eef9466b5817e3bf379078dc1407959c1ff85d6de WatchSource:0}: Error finding container a11310fc561143c1941fda0eef9466b5817e3bf379078dc1407959c1ff85d6de: Status 404 returned error can't find the container with id a11310fc561143c1941fda0eef9466b5817e3bf379078dc1407959c1ff85d6de Jul 15 12:30:00.338986 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.338920 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-99db84ea-29735310-zgk48"] Jul 15 12:30:00.341577 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:30:00.341551 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc39205c_1b2c_4346_b1e6_613cbd9e62d9.slice/crio-4db6662763eea9fc5bdf609984f6f5ba8eb693724d2fdabfb2c6d9c7a171acf6 WatchSource:0}: Error finding container 4db6662763eea9fc5bdf609984f6f5ba8eb693724d2fdabfb2c6d9c7a171acf6: Status 404 returned error can't find the container with id 4db6662763eea9fc5bdf609984f6f5ba8eb693724d2fdabfb2c6d9c7a171acf6 Jul 15 12:30:00.702432 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.702388 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-99db84ea-29735310-zgk48" event={"ID":"fc39205c-1b2c-4346-b1e6-613cbd9e62d9","Type":"ContainerStarted","Data":"df7aa4759d02a0763fc3d98f91c88d31ab3de296b9670574a1174a84f9b370ab"} Jul 15 12:30:00.702618 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.702452 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-99db84ea-29735310-zgk48" event={"ID":"fc39205c-1b2c-4346-b1e6-613cbd9e62d9","Type":"ContainerStarted","Data":"4db6662763eea9fc5bdf609984f6f5ba8eb693724d2fdabfb2c6d9c7a171acf6"} Jul 15 12:30:00.704176 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.704146 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735310-r9kxg" event={"ID":"d8f0d292-b419-4b2f-81de-7c25e974a297","Type":"ContainerStarted","Data":"686e3a63d999d77a6c984eb448be7d53a71498a104dfc846e5ef5f9390f944a2"} Jul 15 12:30:00.704282 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.704195 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735310-r9kxg" event={"ID":"d8f0d292-b419-4b2f-81de-7c25e974a297","Type":"ContainerStarted","Data":"a11310fc561143c1941fda0eef9466b5817e3bf379078dc1407959c1ff85d6de"} Jul 15 12:30:00.717151 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.717083 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-99db84ea-29735310-zgk48" podStartSLOduration=0.717062751 podStartE2EDuration="717.062751ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:30:00.716252017 +0000 UTC m=+1868.129780381" watchObservedRunningTime="2026-07-15 12:30:00.717062751 +0000 UTC m=+1868.130591102" Jul 15 12:30:00.732445 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:00.732356 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735310-r9kxg" podStartSLOduration=0.732341184 podStartE2EDuration="732.341184ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:30:00.730218146 +0000 UTC m=+1868.143746498" watchObservedRunningTime="2026-07-15 12:30:00.732341184 +0000 UTC m=+1868.145869542" Jul 15 12:30:01.710655 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:01.710623 2570 generic.go:356] "Generic (PLEG): container finished" podID="d8f0d292-b419-4b2f-81de-7c25e974a297" containerID="686e3a63d999d77a6c984eb448be7d53a71498a104dfc846e5ef5f9390f944a2" exitCode=0 Jul 15 12:30:01.711122 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:01.710707 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735310-r9kxg" event={"ID":"d8f0d292-b419-4b2f-81de-7c25e974a297","Type":"ContainerDied","Data":"686e3a63d999d77a6c984eb448be7d53a71498a104dfc846e5ef5f9390f944a2"} Jul 15 12:30:01.712091 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:01.712068 2570 generic.go:356] "Generic (PLEG): container finished" podID="fc39205c-1b2c-4346-b1e6-613cbd9e62d9" containerID="df7aa4759d02a0763fc3d98f91c88d31ab3de296b9670574a1174a84f9b370ab" exitCode=0 Jul 15 12:30:01.712201 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:01.712147 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-99db84ea-29735310-zgk48" event={"ID":"fc39205c-1b2c-4346-b1e6-613cbd9e62d9","Type":"ContainerDied","Data":"df7aa4759d02a0763fc3d98f91c88d31ab3de296b9670574a1174a84f9b370ab"} Jul 15 12:30:02.856102 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:02.856077 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-99db84ea-29735310-zgk48" Jul 15 12:30:02.887672 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:02.887646 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735310-r9kxg" Jul 15 12:30:03.724658 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:03.724628 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735310-r9kxg" Jul 15 12:30:03.724658 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:03.724636 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735310-r9kxg" event={"ID":"d8f0d292-b419-4b2f-81de-7c25e974a297","Type":"ContainerDied","Data":"a11310fc561143c1941fda0eef9466b5817e3bf379078dc1407959c1ff85d6de"} Jul 15 12:30:03.724930 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:03.724728 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a11310fc561143c1941fda0eef9466b5817e3bf379078dc1407959c1ff85d6de" Jul 15 12:30:03.726597 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:03.726570 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-99db84ea-29735310-zgk48" event={"ID":"fc39205c-1b2c-4346-b1e6-613cbd9e62d9","Type":"ContainerDied","Data":"4db6662763eea9fc5bdf609984f6f5ba8eb693724d2fdabfb2c6d9c7a171acf6"} Jul 15 12:30:03.726787 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:03.726583 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-99db84ea-29735310-zgk48" Jul 15 12:30:03.726787 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:03.726778 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4db6662763eea9fc5bdf609984f6f5ba8eb693724d2fdabfb2c6d9c7a171acf6" Jul 15 12:30:27.863970 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:27.863934 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w"] Jul 15 12:30:27.864453 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:27.864334 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc39205c-1b2c-4346-b1e6-613cbd9e62d9" containerName="cleanup" Jul 15 12:30:27.864453 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:27.864347 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc39205c-1b2c-4346-b1e6-613cbd9e62d9" containerName="cleanup" Jul 15 12:30:27.864453 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:27.864364 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8f0d292-b419-4b2f-81de-7c25e974a297" containerName="cleanup" Jul 15 12:30:27.864453 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:27.864369 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8f0d292-b419-4b2f-81de-7c25e974a297" containerName="cleanup" Jul 15 12:30:27.864453 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:27.864429 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8f0d292-b419-4b2f-81de-7c25e974a297" containerName="cleanup" Jul 15 12:30:27.864453 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:27.864438 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc39205c-1b2c-4346-b1e6-613cbd9e62d9" containerName="cleanup" Jul 15 12:30:27.867306 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:27.867289 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w" Jul 15 12:30:27.869412 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:27.869392 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tf4tp\"" Jul 15 12:30:27.876684 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:27.876646 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w"] Jul 15 12:30:27.941962 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:27.941931 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ca53125-90e8-42a4-9a8e-f808b0ba425d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w\" (UID: \"6ca53125-90e8-42a4-9a8e-f808b0ba425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w" Jul 15 12:30:28.043118 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:28.043084 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ca53125-90e8-42a4-9a8e-f808b0ba425d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w\" (UID: \"6ca53125-90e8-42a4-9a8e-f808b0ba425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w" Jul 15 12:30:28.043707 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:28.043689 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ca53125-90e8-42a4-9a8e-f808b0ba425d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w\" (UID: \"6ca53125-90e8-42a4-9a8e-f808b0ba425d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w" Jul 15 12:30:28.179563 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:28.179471 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w" Jul 15 12:30:28.342452 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:28.342427 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w"] Jul 15 12:30:28.344817 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:30:28.344789 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca53125_90e8_42a4_9a8e_f808b0ba425d.slice/crio-afb124e50322f7bfd7b5cb914d8d991f3d520801db52ea1c6718bae8dd6c6f39 WatchSource:0}: Error finding container afb124e50322f7bfd7b5cb914d8d991f3d520801db52ea1c6718bae8dd6c6f39: Status 404 returned error can't find the container with id afb124e50322f7bfd7b5cb914d8d991f3d520801db52ea1c6718bae8dd6c6f39 Jul 15 12:30:28.860769 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:28.860713 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w" event={"ID":"6ca53125-90e8-42a4-9a8e-f808b0ba425d","Type":"ContainerStarted","Data":"7c497abaadab0b67daf1307c9c35e63206abdb7541611ae4298cd5f29411b976"} Jul 15 12:30:28.860969 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:28.860818 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w" event={"ID":"6ca53125-90e8-42a4-9a8e-f808b0ba425d","Type":"ContainerStarted","Data":"afb124e50322f7bfd7b5cb914d8d991f3d520801db52ea1c6718bae8dd6c6f39"} Jul 15 12:30:28.875893 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:28.875843 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w" podStartSLOduration=1.875825464 podStartE2EDuration="1.875825464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:30:28.874636411 +0000 UTC m=+1896.288164762" watchObservedRunningTime="2026-07-15 12:30:28.875825464 +0000 UTC m=+1896.289353816" Jul 15 12:30:29.866971 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:29.866936 2570 generic.go:356] "Generic (PLEG): container finished" podID="6ca53125-90e8-42a4-9a8e-f808b0ba425d" containerID="7c497abaadab0b67daf1307c9c35e63206abdb7541611ae4298cd5f29411b976" exitCode=0 Jul 15 12:30:29.867146 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:29.866982 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w" event={"ID":"6ca53125-90e8-42a4-9a8e-f808b0ba425d","Type":"ContainerDied","Data":"7c497abaadab0b67daf1307c9c35e63206abdb7541611ae4298cd5f29411b976"} Jul 15 12:30:31.004259 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:31.004230 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w" Jul 15 12:30:31.068547 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:31.068509 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ca53125-90e8-42a4-9a8e-f808b0ba425d-maas-api-service-ca\") pod \"6ca53125-90e8-42a4-9a8e-f808b0ba425d\" (UID: \"6ca53125-90e8-42a4-9a8e-f808b0ba425d\") " Jul 15 12:30:31.068896 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:31.068875 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca53125-90e8-42a4-9a8e-f808b0ba425d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ca53125-90e8-42a4-9a8e-f808b0ba425d" (UID: "6ca53125-90e8-42a4-9a8e-f808b0ba425d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:30:31.170080 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:31.169989 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ca53125-90e8-42a4-9a8e-f808b0ba425d-maas-api-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:30:31.878306 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:31.878267 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w" event={"ID":"6ca53125-90e8-42a4-9a8e-f808b0ba425d","Type":"ContainerDied","Data":"afb124e50322f7bfd7b5cb914d8d991f3d520801db52ea1c6718bae8dd6c6f39"} Jul 15 12:30:31.878306 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:31.878301 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w" Jul 15 12:30:31.878521 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:31.878327 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afb124e50322f7bfd7b5cb914d8d991f3d520801db52ea1c6718bae8dd6c6f39" Jul 15 12:30:38.361976 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.361931 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc"] Jul 15 12:30:38.362552 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.362189 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" podUID="8db1effd-21ff-4206-a038-922a6318afc4" containerName="maas-api" containerID="cri-o://2ba5e624aca7fe4b3de64131c84b057d7f57df5dbbfa114830b1126388656091" gracePeriod=30 Jul 15 12:30:38.389319 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.389285 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-99db84ea-29735310-zgk48"] Jul 15 12:30:38.393931 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.393905 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-99db84ea-29735310-zgk48"] Jul 15 12:30:38.422759 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.422699 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:30:38.610044 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.610021 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:30:38.641805 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.641695 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8db1effd-21ff-4206-a038-922a6318afc4-maas-api-tls\") pod \"8db1effd-21ff-4206-a038-922a6318afc4\" (UID: \"8db1effd-21ff-4206-a038-922a6318afc4\") " Jul 15 12:30:38.641805 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.641785 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxkcw\" (UniqueName: \"kubernetes.io/projected/8db1effd-21ff-4206-a038-922a6318afc4-kube-api-access-zxkcw\") pod \"8db1effd-21ff-4206-a038-922a6318afc4\" (UID: \"8db1effd-21ff-4206-a038-922a6318afc4\") " Jul 15 12:30:38.642040 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.641813 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db1effd-21ff-4206-a038-922a6318afc4-openshift-service-ca\") pod \"8db1effd-21ff-4206-a038-922a6318afc4\" (UID: \"8db1effd-21ff-4206-a038-922a6318afc4\") " Jul 15 12:30:38.642285 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.642255 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db1effd-21ff-4206-a038-922a6318afc4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8db1effd-21ff-4206-a038-922a6318afc4" (UID: "8db1effd-21ff-4206-a038-922a6318afc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:30:38.644133 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.644100 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db1effd-21ff-4206-a038-922a6318afc4-kube-api-access-zxkcw" (OuterVolumeSpecName: "kube-api-access-zxkcw") pod "8db1effd-21ff-4206-a038-922a6318afc4" (UID: "8db1effd-21ff-4206-a038-922a6318afc4"). InnerVolumeSpecName "kube-api-access-zxkcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:30:38.644252 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.644147 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8db1effd-21ff-4206-a038-922a6318afc4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8db1effd-21ff-4206-a038-922a6318afc4" (UID: "8db1effd-21ff-4206-a038-922a6318afc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:30:38.742934 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.742896 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8db1effd-21ff-4206-a038-922a6318afc4-maas-api-tls\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:30:38.742934 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.742927 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxkcw\" (UniqueName: \"kubernetes.io/projected/8db1effd-21ff-4206-a038-922a6318afc4-kube-api-access-zxkcw\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:30:38.742934 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.742938 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db1effd-21ff-4206-a038-922a6318afc4-openshift-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:30:38.914008 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.913916 2570 generic.go:356] "Generic (PLEG): container finished" podID="8db1effd-21ff-4206-a038-922a6318afc4" containerID="2ba5e624aca7fe4b3de64131c84b057d7f57df5dbbfa114830b1126388656091" exitCode=0 Jul 15 12:30:38.914008 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.913958 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" Jul 15 12:30:38.914008 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.913985 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" event={"ID":"8db1effd-21ff-4206-a038-922a6318afc4","Type":"ContainerDied","Data":"2ba5e624aca7fe4b3de64131c84b057d7f57df5dbbfa114830b1126388656091"} Jul 15 12:30:38.914232 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.914029 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc" event={"ID":"8db1effd-21ff-4206-a038-922a6318afc4","Type":"ContainerDied","Data":"57c64e42435f65ec6fcb237f341fa29c57261bce1d1432711a654165be15864b"} Jul 15 12:30:38.914232 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.914051 2570 scope.go:117] "RemoveContainer" containerID="2ba5e624aca7fe4b3de64131c84b057d7f57df5dbbfa114830b1126388656091" Jul 15 12:30:38.923551 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.923529 2570 scope.go:117] "RemoveContainer" containerID="2ba5e624aca7fe4b3de64131c84b057d7f57df5dbbfa114830b1126388656091" Jul 15 12:30:38.923819 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:30:38.923800 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba5e624aca7fe4b3de64131c84b057d7f57df5dbbfa114830b1126388656091\": container with ID starting with 2ba5e624aca7fe4b3de64131c84b057d7f57df5dbbfa114830b1126388656091 not found: ID does not exist" containerID="2ba5e624aca7fe4b3de64131c84b057d7f57df5dbbfa114830b1126388656091" Jul 15 12:30:38.923865 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.923830 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba5e624aca7fe4b3de64131c84b057d7f57df5dbbfa114830b1126388656091"} err="failed to get container status \"2ba5e624aca7fe4b3de64131c84b057d7f57df5dbbfa114830b1126388656091\": rpc error: code = NotFound desc = could not find container \"2ba5e624aca7fe4b3de64131c84b057d7f57df5dbbfa114830b1126388656091\": container with ID starting with 2ba5e624aca7fe4b3de64131c84b057d7f57df5dbbfa114830b1126388656091 not found: ID does not exist" Jul 15 12:30:38.935089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.935054 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc"] Jul 15 12:30:38.938843 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:38.938821 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-99db84ea-56bd768c5-z5zkc"] Jul 15 12:30:39.092500 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:39.092463 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8db1effd-21ff-4206-a038-922a6318afc4" path="/var/lib/kubelet/pods/8db1effd-21ff-4206-a038-922a6318afc4/volumes" Jul 15 12:30:39.092797 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:39.092784 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc39205c-1b2c-4346-b1e6-613cbd9e62d9" path="/var/lib/kubelet/pods/fc39205c-1b2c-4346-b1e6-613cbd9e62d9/volumes" Jul 15 12:30:42.959256 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:42.959207 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:30:48.965447 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:48.965400 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:30:49.035962 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:49.035886 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:30:53.137288 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:53.137257 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp"] Jul 15 12:30:53.137718 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:53.137447 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" podUID="7e867d7c-fd95-4283-858c-4e843a136781" containerName="istio-proxy" containerID="cri-o://51673f9fc48a14c574aba40c0fea13ce0f1e35a9269372df49f3c2e22be41e20" gracePeriod=30 Jul 15 12:30:53.169677 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:53.169644 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:30:53.809785 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:53.809734 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:30:53.909196 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:53.909165 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:30:54.009570 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:54.009532 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:30:55.573766 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:55.573701 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:30:58.389609 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.389582 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:30:58.522333 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.522300 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-istio-envoy\") pod \"7e867d7c-fd95-4283-858c-4e843a136781\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " Jul 15 12:30:58.522504 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.522388 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-istio-data\") pod \"7e867d7c-fd95-4283-858c-4e843a136781\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " Jul 15 12:30:58.522504 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.522413 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e867d7c-fd95-4283-858c-4e843a136781-istiod-ca-cert\") pod \"7e867d7c-fd95-4283-858c-4e843a136781\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " Jul 15 12:30:58.522504 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.522438 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-credential-socket\") pod \"7e867d7c-fd95-4283-858c-4e843a136781\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " Jul 15 12:30:58.522504 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.522468 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-workload-certs\") pod \"7e867d7c-fd95-4283-858c-4e843a136781\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " Jul 15 12:30:58.522504 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.522499 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxj6b\" (UniqueName: \"kubernetes.io/projected/7e867d7c-fd95-4283-858c-4e843a136781-kube-api-access-pxj6b\") pod \"7e867d7c-fd95-4283-858c-4e843a136781\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " Jul 15 12:30:58.522780 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.522535 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e867d7c-fd95-4283-858c-4e843a136781-istio-podinfo\") pod \"7e867d7c-fd95-4283-858c-4e843a136781\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " Jul 15 12:30:58.522780 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.522567 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-workload-socket\") pod \"7e867d7c-fd95-4283-858c-4e843a136781\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " Jul 15 12:30:58.522780 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.522599 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e867d7c-fd95-4283-858c-4e843a136781-istio-token\") pod \"7e867d7c-fd95-4283-858c-4e843a136781\" (UID: \"7e867d7c-fd95-4283-858c-4e843a136781\") " Jul 15 12:30:58.522938 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.522808 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e867d7c-fd95-4283-858c-4e843a136781-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e867d7c-fd95-4283-858c-4e843a136781" (UID: "7e867d7c-fd95-4283-858c-4e843a136781"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:30:58.522938 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.522808 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e867d7c-fd95-4283-858c-4e843a136781" (UID: "7e867d7c-fd95-4283-858c-4e843a136781"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:30:58.523043 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.522955 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e867d7c-fd95-4283-858c-4e843a136781-istiod-ca-cert\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:30:58.523043 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.522976 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-workload-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:30:58.523241 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.523161 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e867d7c-fd95-4283-858c-4e843a136781" (UID: "7e867d7c-fd95-4283-858c-4e843a136781"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:30:58.523241 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.523204 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e867d7c-fd95-4283-858c-4e843a136781" (UID: "7e867d7c-fd95-4283-858c-4e843a136781"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:30:58.523393 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.523367 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e867d7c-fd95-4283-858c-4e843a136781" (UID: "7e867d7c-fd95-4283-858c-4e843a136781"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:30:58.525045 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.525015 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e867d7c-fd95-4283-858c-4e843a136781-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e867d7c-fd95-4283-858c-4e843a136781" (UID: "7e867d7c-fd95-4283-858c-4e843a136781"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:30:58.525045 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.525028 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e867d7c-fd95-4283-858c-4e843a136781-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e867d7c-fd95-4283-858c-4e843a136781" (UID: "7e867d7c-fd95-4283-858c-4e843a136781"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:30:58.525224 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.525200 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e867d7c-fd95-4283-858c-4e843a136781-kube-api-access-pxj6b" (OuterVolumeSpecName: "kube-api-access-pxj6b") pod "7e867d7c-fd95-4283-858c-4e843a136781" (UID: "7e867d7c-fd95-4283-858c-4e843a136781"). InnerVolumeSpecName "kube-api-access-pxj6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:30:58.525283 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.525265 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e867d7c-fd95-4283-858c-4e843a136781" (UID: "7e867d7c-fd95-4283-858c-4e843a136781"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:30:58.624026 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.623985 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-istio-data\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:30:58.624026 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.624018 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-credential-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:30:58.624026 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.624029 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxj6b\" (UniqueName: \"kubernetes.io/projected/7e867d7c-fd95-4283-858c-4e843a136781-kube-api-access-pxj6b\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:30:58.624272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.624039 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e867d7c-fd95-4283-858c-4e843a136781-istio-podinfo\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:30:58.624272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.624048 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-workload-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:30:58.624272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.624056 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e867d7c-fd95-4283-858c-4e843a136781-istio-token\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:30:58.624272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:58.624064 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e867d7c-fd95-4283-858c-4e843a136781-istio-envoy\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:30:59.021485 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:59.021447 2570 generic.go:356] "Generic (PLEG): container finished" podID="7e867d7c-fd95-4283-858c-4e843a136781" containerID="51673f9fc48a14c574aba40c0fea13ce0f1e35a9269372df49f3c2e22be41e20" exitCode=0 Jul 15 12:30:59.021696 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:59.021497 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" Jul 15 12:30:59.021696 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:59.021536 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" event={"ID":"7e867d7c-fd95-4283-858c-4e843a136781","Type":"ContainerDied","Data":"51673f9fc48a14c574aba40c0fea13ce0f1e35a9269372df49f3c2e22be41e20"} Jul 15 12:30:59.021696 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:59.021607 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp" event={"ID":"7e867d7c-fd95-4283-858c-4e843a136781","Type":"ContainerDied","Data":"18ebc261d2ae16e0eb4fab8c26b8c97051288476b01cd1fb06964cd1b2cbceb1"} Jul 15 12:30:59.021696 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:59.021640 2570 scope.go:117] "RemoveContainer" containerID="51673f9fc48a14c574aba40c0fea13ce0f1e35a9269372df49f3c2e22be41e20" Jul 15 12:30:59.031190 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:59.031170 2570 scope.go:117] "RemoveContainer" containerID="262ef56128a15e934d127a0720bb2083d6f1aaabc6cfebbc772682bb8c729d7f" Jul 15 12:30:59.039487 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:59.039467 2570 scope.go:117] "RemoveContainer" containerID="51673f9fc48a14c574aba40c0fea13ce0f1e35a9269372df49f3c2e22be41e20" Jul 15 12:30:59.039727 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:30:59.039710 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51673f9fc48a14c574aba40c0fea13ce0f1e35a9269372df49f3c2e22be41e20\": container with ID starting with 51673f9fc48a14c574aba40c0fea13ce0f1e35a9269372df49f3c2e22be41e20 not found: ID does not exist" containerID="51673f9fc48a14c574aba40c0fea13ce0f1e35a9269372df49f3c2e22be41e20" Jul 15 12:30:59.039796 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:59.039770 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51673f9fc48a14c574aba40c0fea13ce0f1e35a9269372df49f3c2e22be41e20"} err="failed to get container status \"51673f9fc48a14c574aba40c0fea13ce0f1e35a9269372df49f3c2e22be41e20\": rpc error: code = NotFound desc = could not find container \"51673f9fc48a14c574aba40c0fea13ce0f1e35a9269372df49f3c2e22be41e20\": container with ID starting with 51673f9fc48a14c574aba40c0fea13ce0f1e35a9269372df49f3c2e22be41e20 not found: ID does not exist" Jul 15 12:30:59.039796 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:59.039791 2570 scope.go:117] "RemoveContainer" containerID="262ef56128a15e934d127a0720bb2083d6f1aaabc6cfebbc772682bb8c729d7f" Jul 15 12:30:59.040011 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:30:59.039995 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"262ef56128a15e934d127a0720bb2083d6f1aaabc6cfebbc772682bb8c729d7f\": container with ID starting with 262ef56128a15e934d127a0720bb2083d6f1aaabc6cfebbc772682bb8c729d7f not found: ID does not exist" containerID="262ef56128a15e934d127a0720bb2083d6f1aaabc6cfebbc772682bb8c729d7f" Jul 15 12:30:59.040053 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:59.040016 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"262ef56128a15e934d127a0720bb2083d6f1aaabc6cfebbc772682bb8c729d7f"} err="failed to get container status \"262ef56128a15e934d127a0720bb2083d6f1aaabc6cfebbc772682bb8c729d7f\": rpc error: code = NotFound desc = could not find container \"262ef56128a15e934d127a0720bb2083d6f1aaabc6cfebbc772682bb8c729d7f\": container with ID starting with 262ef56128a15e934d127a0720bb2083d6f1aaabc6cfebbc772682bb8c729d7f not found: ID does not exist" Jul 15 12:30:59.044385 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:59.044362 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp"] Jul 15 12:30:59.050712 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:59.050693 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-99db84ea-openshift-default-c9587dbcf-fjrqp"] Jul 15 12:30:59.092886 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:30:59.092850 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e867d7c-fd95-4283-858c-4e843a136781" path="/var/lib/kubelet/pods/7e867d7c-fd95-4283-858c-4e843a136781/volumes" Jul 15 12:31:00.061933 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:31:00.061879 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:31:02.008026 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:31:02.007983 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:31:02.113989 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:31:02.113942 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:31:02.207715 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:31:02.207684 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:31:02.313070 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:31:02.312979 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:31:02.787380 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:31:02.787327 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:31:17.860669 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:31:17.860623 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:31:28.850579 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:31:28.850520 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:31:34.839391 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:31:34.839347 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:31:34.932981 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:31:34.932946 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:31:35.047330 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:31:35.047292 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:31:51.092771 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:31:51.092638 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:31:53.294303 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:31:53.294266 2570 scope.go:117] "RemoveContainer" containerID="bede73d7fa1e34fcba3b90c1736801e8d942234d027ad01301ae49a73b6fe36a" Jul 15 12:31:57.464064 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:31:57.464019 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:31:57.564359 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:31:57.564327 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:09.670558 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:09.670512 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:09.770612 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:09.770570 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:14.832590 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:14.832545 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:14.913006 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:14.912970 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:15.007938 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:15.007901 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:15.107284 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:15.107248 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:15.939997 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:15.939958 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:16.036167 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:16.036122 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:18.039836 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:18.039799 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:18.140262 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:18.140227 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:24.210031 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:24.209997 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:24.287661 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:24.287622 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:24.654896 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:24.654849 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:30.946678 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:30.946640 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:31.026409 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.026370 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:31.273689 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.273607 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc"] Jul 15 12:32:31.273987 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.273974 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e867d7c-fd95-4283-858c-4e843a136781" containerName="istio-proxy" Jul 15 12:32:31.274049 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.273990 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e867d7c-fd95-4283-858c-4e843a136781" containerName="istio-proxy" Jul 15 12:32:31.274049 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.274006 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca53125-90e8-42a4-9a8e-f808b0ba425d" containerName="revoke-keys" Jul 15 12:32:31.274049 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.274012 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca53125-90e8-42a4-9a8e-f808b0ba425d" containerName="revoke-keys" Jul 15 12:32:31.274049 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.274027 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db1effd-21ff-4206-a038-922a6318afc4" containerName="maas-api" Jul 15 12:32:31.274049 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.274033 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db1effd-21ff-4206-a038-922a6318afc4" containerName="maas-api" Jul 15 12:32:31.274210 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.274094 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ca53125-90e8-42a4-9a8e-f808b0ba425d" containerName="revoke-keys" Jul 15 12:32:31.274210 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.274103 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e867d7c-fd95-4283-858c-4e843a136781" containerName="istio-proxy" Jul 15 12:32:31.274210 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.274110 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e867d7c-fd95-4283-858c-4e843a136781" containerName="istio-proxy" Jul 15 12:32:31.274210 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.274117 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db1effd-21ff-4206-a038-922a6318afc4" containerName="maas-api" Jul 15 12:32:31.276241 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.276219 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.278293 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.278272 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d3e5d67b-openshift-default-dockercfg-wszm7\"" Jul 15 12:32:31.278411 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.278291 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 12:32:31.297969 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.297930 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:31.298913 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.298891 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc"] Jul 15 12:32:31.376858 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.376823 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-token\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.376858 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.376859 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-credential-socket\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.377068 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.376888 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-workload-certs\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.377068 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.376906 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istiod-ca-cert\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.377068 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.376954 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-workload-socket\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.377068 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.376969 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lccd\" (UniqueName: \"kubernetes.io/projected/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-kube-api-access-2lccd\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.377068 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.377050 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-envoy\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.377231 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.377094 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-data\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.377231 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.377123 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-podinfo\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.477852 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.477805 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-token\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.477852 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.477849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-credential-socket\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.478060 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.477976 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-workload-certs\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.478060 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.478009 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istiod-ca-cert\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.478060 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.478059 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-workload-socket\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.478182 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.478075 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lccd\" (UniqueName: \"kubernetes.io/projected/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-kube-api-access-2lccd\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.478182 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.478126 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-envoy\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.478182 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.478158 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-data\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.478393 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.478248 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-credential-socket\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.478393 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.478294 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-podinfo\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.478393 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.478318 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-workload-certs\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.478569 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.478478 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-workload-socket\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.478569 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.478520 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-data\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.478782 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.478761 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istiod-ca-cert\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.480626 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.480599 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-envoy\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.480864 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.480844 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-podinfo\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.486173 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.486145 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-token\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.486260 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.486226 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lccd\" (UniqueName: \"kubernetes.io/projected/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-kube-api-access-2lccd\") pod \"e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.499023 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.498995 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:31.592651 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.592560 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:31.753994 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.753964 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc"] Jul 15 12:32:31.756548 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:32:31.756519 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f96e492_69ef_4bd0_bd29_f59e93b4cdb3.slice/crio-cf1bbd6a6b0f91f517eb7dd9539f5cc858342061521ee80613b3230f65260f66 WatchSource:0}: Error finding container cf1bbd6a6b0f91f517eb7dd9539f5cc858342061521ee80613b3230f65260f66: Status 404 returned error can't find the container with id cf1bbd6a6b0f91f517eb7dd9539f5cc858342061521ee80613b3230f65260f66 Jul 15 12:32:31.758805 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.758761 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:32:31.758874 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.758860 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:32:31.758925 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:31.758909 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:32:32.516255 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:32.516216 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" event={"ID":"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3","Type":"ContainerStarted","Data":"0867399d7431a52ce29ee66d91f1919af7e1f584440e16db2dd984521158727c"} Jul 15 12:32:32.516734 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:32.516270 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" event={"ID":"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3","Type":"ContainerStarted","Data":"cf1bbd6a6b0f91f517eb7dd9539f5cc858342061521ee80613b3230f65260f66"} Jul 15 12:32:32.543159 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:32.543103 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" podStartSLOduration=1.5430873410000001 podStartE2EDuration="1.543087341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:32:32.541402568 +0000 UTC m=+2019.954930920" watchObservedRunningTime="2026-07-15 12:32:32.543087341 +0000 UTC m=+2019.956615762" Jul 15 12:32:32.593661 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:32.593625 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:32.598561 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:32.598535 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:33.521575 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:33.521543 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:33.522634 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:33.522614 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:34.517913 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:34.517872 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:34.619536 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:34.619502 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:34.723453 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:34.723419 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f"] Jul 15 12:32:34.724016 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:34.723991 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e867d7c-fd95-4283-858c-4e843a136781" containerName="istio-proxy" Jul 15 12:32:34.724202 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:34.724187 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e867d7c-fd95-4283-858c-4e843a136781" containerName="istio-proxy" Jul 15 12:32:34.727032 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:34.727011 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" Jul 15 12:32:34.728933 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:34.728914 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d3e5d67b-serving-cert\"" Jul 15 12:32:34.739374 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:34.739350 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f"] Jul 15 12:32:34.769489 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:34.769411 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:34.911885 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:34.911848 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4c57401-6126-49a1-bba7-ce344ac1014f-maas-api-tls\") pod \"maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f\" (UID: \"b4c57401-6126-49a1-bba7-ce344ac1014f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" Jul 15 12:32:34.911885 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:34.911889 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg8pk\" (UniqueName: \"kubernetes.io/projected/b4c57401-6126-49a1-bba7-ce344ac1014f-kube-api-access-bg8pk\") pod \"maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f\" (UID: \"b4c57401-6126-49a1-bba7-ce344ac1014f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" Jul 15 12:32:34.912150 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:34.912014 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c57401-6126-49a1-bba7-ce344ac1014f-openshift-service-ca\") pod \"maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f\" (UID: \"b4c57401-6126-49a1-bba7-ce344ac1014f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" Jul 15 12:32:35.013208 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:35.013166 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4c57401-6126-49a1-bba7-ce344ac1014f-maas-api-tls\") pod \"maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f\" (UID: \"b4c57401-6126-49a1-bba7-ce344ac1014f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" Jul 15 12:32:35.013208 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:35.013211 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg8pk\" (UniqueName: \"kubernetes.io/projected/b4c57401-6126-49a1-bba7-ce344ac1014f-kube-api-access-bg8pk\") pod \"maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f\" (UID: \"b4c57401-6126-49a1-bba7-ce344ac1014f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" Jul 15 12:32:35.013416 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:35.013267 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c57401-6126-49a1-bba7-ce344ac1014f-openshift-service-ca\") pod \"maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f\" (UID: \"b4c57401-6126-49a1-bba7-ce344ac1014f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" Jul 15 12:32:35.013953 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:35.013932 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c57401-6126-49a1-bba7-ce344ac1014f-openshift-service-ca\") pod \"maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f\" (UID: \"b4c57401-6126-49a1-bba7-ce344ac1014f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" Jul 15 12:32:35.015789 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:35.015761 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4c57401-6126-49a1-bba7-ce344ac1014f-maas-api-tls\") pod \"maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f\" (UID: \"b4c57401-6126-49a1-bba7-ce344ac1014f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" Jul 15 12:32:35.017989 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:35.017962 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:35.020983 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:35.020929 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg8pk\" (UniqueName: \"kubernetes.io/projected/b4c57401-6126-49a1-bba7-ce344ac1014f-kube-api-access-bg8pk\") pod \"maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f\" (UID: \"b4c57401-6126-49a1-bba7-ce344ac1014f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" Jul 15 12:32:35.040914 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:35.040875 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" Jul 15 12:32:35.188407 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:35.188375 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f"] Jul 15 12:32:35.190925 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:32:35.190891 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c57401_6126_49a1_bba7_ce344ac1014f.slice/crio-d93cb33b83c45b49b6727c1bb169972b0dbdc330891d3ba01353326d9dd244a0 WatchSource:0}: Error finding container d93cb33b83c45b49b6727c1bb169972b0dbdc330891d3ba01353326d9dd244a0: Status 404 returned error can't find the container with id d93cb33b83c45b49b6727c1bb169972b0dbdc330891d3ba01353326d9dd244a0 Jul 15 12:32:35.192354 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:35.192331 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:32:35.297614 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:35.297507 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:35.400583 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:35.400540 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:35.532004 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:35.531971 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" event={"ID":"b4c57401-6126-49a1-bba7-ce344ac1014f","Type":"ContainerStarted","Data":"d93cb33b83c45b49b6727c1bb169972b0dbdc330891d3ba01353326d9dd244a0"} Jul 15 12:32:36.538833 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:36.538797 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" event={"ID":"b4c57401-6126-49a1-bba7-ce344ac1014f","Type":"ContainerStarted","Data":"bd647846204296901aad949ad508b52a6702f05dfc78fcd04227a9b549f84647"} Jul 15 12:32:36.539259 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:36.538933 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" Jul 15 12:32:36.556579 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:36.556534 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" podStartSLOduration=2.130018507 podStartE2EDuration="2.556519763s" totalImagesPullingTime="426.501256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:32:36.554994408 +0000 UTC m=+2023.968522759" watchObservedRunningTime="2026-07-15 12:32:36.556519763 +0000 UTC m=+2023.970048113" Jul 15 12:32:39.482522 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:39.482487 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:40.179665 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:40.179632 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:40.279998 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:40.279957 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:40.947394 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:40.947356 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:42.549919 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:42.549890 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" Jul 15 12:32:47.345019 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:47.344989 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:47.444397 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:47.444370 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:47.615954 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:47.615863 2570 generic.go:356] "Generic (PLEG): container finished" podID="2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" containerID="0867399d7431a52ce29ee66d91f1919af7e1f584440e16db2dd984521158727c" exitCode=0 Jul 15 12:32:47.616124 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:47.615951 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" event={"ID":"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3","Type":"ContainerDied","Data":"0867399d7431a52ce29ee66d91f1919af7e1f584440e16db2dd984521158727c"} Jul 15 12:32:47.616574 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:47.616555 2570 scope.go:117] "RemoveContainer" containerID="0867399d7431a52ce29ee66d91f1919af7e1f584440e16db2dd984521158727c" Jul 15 12:32:47.617799 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:47.617763 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:32:47.617914 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:47.617856 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:32:47.617914 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:47.617901 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:32:47.715451 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:47.715400 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4"] Jul 15 12:32:47.718799 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:47.718773 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4" Jul 15 12:32:47.720862 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:47.720828 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tf4tp\"" Jul 15 12:32:47.726086 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:47.726010 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4"] Jul 15 12:32:47.751327 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:47.751281 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28edefef-bbd5-4399-91e3-9f917d2bda52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4\" (UID: \"28edefef-bbd5-4399-91e3-9f917d2bda52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4" Jul 15 12:32:47.852920 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:47.852875 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28edefef-bbd5-4399-91e3-9f917d2bda52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4\" (UID: \"28edefef-bbd5-4399-91e3-9f917d2bda52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4" Jul 15 12:32:47.853841 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:47.853817 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28edefef-bbd5-4399-91e3-9f917d2bda52-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4\" (UID: \"28edefef-bbd5-4399-91e3-9f917d2bda52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4" Jul 15 12:32:48.033142 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:48.033044 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4" Jul 15 12:32:48.179875 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:48.179841 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4"] Jul 15 12:32:48.181722 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:32:48.181693 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28edefef_bbd5_4399_91e3_9f917d2bda52.slice/crio-131d03ada62017e602c7efabc6458835468da995e2b0f7617f37c580615dd290 WatchSource:0}: Error finding container 131d03ada62017e602c7efabc6458835468da995e2b0f7617f37c580615dd290: Status 404 returned error can't find the container with id 131d03ada62017e602c7efabc6458835468da995e2b0f7617f37c580615dd290 Jul 15 12:32:48.522364 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:48.522317 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:48.623681 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:48.623644 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4" event={"ID":"28edefef-bbd5-4399-91e3-9f917d2bda52","Type":"ContainerStarted","Data":"e6460e9a180c852f40cf42ba22681fbf0411c3e492e0a3ed05aeb5db197731ed"} Jul 15 12:32:48.623909 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:48.623714 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4" event={"ID":"28edefef-bbd5-4399-91e3-9f917d2bda52","Type":"ContainerStarted","Data":"131d03ada62017e602c7efabc6458835468da995e2b0f7617f37c580615dd290"} Jul 15 12:32:48.625668 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:48.625641 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" event={"ID":"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3","Type":"ContainerStarted","Data":"1cd23cfa7e4153e4bbf1fbac9704e421fe3a18281f845ac67edca8ac1b4b6a24"} Jul 15 12:32:48.638701 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:48.638557 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4" podStartSLOduration=1.6385349439999999 podStartE2EDuration="1.638534944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:32:48.637885067 +0000 UTC m=+2036.051413440" watchObservedRunningTime="2026-07-15 12:32:48.638534944 +0000 UTC m=+2036.052063298" Jul 15 12:32:49.593672 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:49.593562 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:49.594972 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:49.594943 2570 patch_prober.go:28] interesting pod/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 15 12:32:49.595092 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:49.595007 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" podUID="2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 15 12:32:49.632449 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:49.632413 2570 generic.go:356] "Generic (PLEG): container finished" podID="28edefef-bbd5-4399-91e3-9f917d2bda52" containerID="e6460e9a180c852f40cf42ba22681fbf0411c3e492e0a3ed05aeb5db197731ed" exitCode=0 Jul 15 12:32:49.632627 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:49.632484 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4" event={"ID":"28edefef-bbd5-4399-91e3-9f917d2bda52","Type":"ContainerDied","Data":"e6460e9a180c852f40cf42ba22681fbf0411c3e492e0a3ed05aeb5db197731ed"} Jul 15 12:32:50.593836 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:50.593796 2570 patch_prober.go:28] interesting pod/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 15 12:32:50.594313 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:50.593860 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" podUID="2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 15 12:32:50.811477 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:50.811455 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4" Jul 15 12:32:50.889182 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:50.889139 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28edefef-bbd5-4399-91e3-9f917d2bda52-maas-api-service-ca\") pod \"28edefef-bbd5-4399-91e3-9f917d2bda52\" (UID: \"28edefef-bbd5-4399-91e3-9f917d2bda52\") " Jul 15 12:32:50.889539 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:50.889509 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28edefef-bbd5-4399-91e3-9f917d2bda52-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "28edefef-bbd5-4399-91e3-9f917d2bda52" (UID: "28edefef-bbd5-4399-91e3-9f917d2bda52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:32:50.990551 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:50.990505 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28edefef-bbd5-4399-91e3-9f917d2bda52-maas-api-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:32:51.593852 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:51.593813 2570 patch_prober.go:28] interesting pod/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 15 12:32:51.594270 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:51.593874 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" podUID="2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 15 12:32:51.643617 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:51.643584 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4" Jul 15 12:32:51.643817 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:51.643625 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4" event={"ID":"28edefef-bbd5-4399-91e3-9f917d2bda52","Type":"ContainerDied","Data":"131d03ada62017e602c7efabc6458835468da995e2b0f7617f37c580615dd290"} Jul 15 12:32:51.643817 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:51.643674 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="131d03ada62017e602c7efabc6458835468da995e2b0f7617f37c580615dd290" Jul 15 12:32:52.593977 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:52.593931 2570 patch_prober.go:28] interesting pod/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 15 12:32:52.594458 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:52.593997 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" podUID="2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 15 12:32:53.597602 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:53.597538 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:53.598107 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:53.598087 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:53.598818 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:53.598802 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:32:58.219859 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.219812 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f"] Jul 15 12:32:58.220395 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.220075 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" podUID="b4c57401-6126-49a1-bba7-ce344ac1014f" containerName="maas-api" containerID="cri-o://bd647846204296901aad949ad508b52a6702f05dfc78fcd04227a9b549f84647" gracePeriod=30 Jul 15 12:32:58.277769 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.277708 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:32:58.480048 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.479962 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" Jul 15 12:32:58.560011 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.559970 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bg8pk\" (UniqueName: \"kubernetes.io/projected/b4c57401-6126-49a1-bba7-ce344ac1014f-kube-api-access-bg8pk\") pod \"b4c57401-6126-49a1-bba7-ce344ac1014f\" (UID: \"b4c57401-6126-49a1-bba7-ce344ac1014f\") " Jul 15 12:32:58.560227 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.560083 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c57401-6126-49a1-bba7-ce344ac1014f-openshift-service-ca\") pod \"b4c57401-6126-49a1-bba7-ce344ac1014f\" (UID: \"b4c57401-6126-49a1-bba7-ce344ac1014f\") " Jul 15 12:32:58.560227 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.560116 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4c57401-6126-49a1-bba7-ce344ac1014f-maas-api-tls\") pod \"b4c57401-6126-49a1-bba7-ce344ac1014f\" (UID: \"b4c57401-6126-49a1-bba7-ce344ac1014f\") " Jul 15 12:32:58.560468 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.560443 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c57401-6126-49a1-bba7-ce344ac1014f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b4c57401-6126-49a1-bba7-ce344ac1014f" (UID: "b4c57401-6126-49a1-bba7-ce344ac1014f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:32:58.562420 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.562385 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c57401-6126-49a1-bba7-ce344ac1014f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b4c57401-6126-49a1-bba7-ce344ac1014f" (UID: "b4c57401-6126-49a1-bba7-ce344ac1014f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:32:58.562510 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.562411 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c57401-6126-49a1-bba7-ce344ac1014f-kube-api-access-bg8pk" (OuterVolumeSpecName: "kube-api-access-bg8pk") pod "b4c57401-6126-49a1-bba7-ce344ac1014f" (UID: "b4c57401-6126-49a1-bba7-ce344ac1014f"). InnerVolumeSpecName "kube-api-access-bg8pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:32:58.661200 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.661154 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4c57401-6126-49a1-bba7-ce344ac1014f-openshift-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:32:58.661200 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.661193 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4c57401-6126-49a1-bba7-ce344ac1014f-maas-api-tls\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:32:58.661200 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.661207 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg8pk\" (UniqueName: \"kubernetes.io/projected/b4c57401-6126-49a1-bba7-ce344ac1014f-kube-api-access-bg8pk\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:32:58.686153 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.686115 2570 generic.go:356] "Generic (PLEG): container finished" podID="b4c57401-6126-49a1-bba7-ce344ac1014f" containerID="bd647846204296901aad949ad508b52a6702f05dfc78fcd04227a9b549f84647" exitCode=0 Jul 15 12:32:58.686354 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.686179 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" event={"ID":"b4c57401-6126-49a1-bba7-ce344ac1014f","Type":"ContainerDied","Data":"bd647846204296901aad949ad508b52a6702f05dfc78fcd04227a9b549f84647"} Jul 15 12:32:58.686354 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.686225 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" event={"ID":"b4c57401-6126-49a1-bba7-ce344ac1014f","Type":"ContainerDied","Data":"d93cb33b83c45b49b6727c1bb169972b0dbdc330891d3ba01353326d9dd244a0"} Jul 15 12:32:58.686354 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.686247 2570 scope.go:117] "RemoveContainer" containerID="bd647846204296901aad949ad508b52a6702f05dfc78fcd04227a9b549f84647" Jul 15 12:32:58.686354 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.686179 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f" Jul 15 12:32:58.696706 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.696682 2570 scope.go:117] "RemoveContainer" containerID="bd647846204296901aad949ad508b52a6702f05dfc78fcd04227a9b549f84647" Jul 15 12:32:58.697071 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:32:58.697052 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd647846204296901aad949ad508b52a6702f05dfc78fcd04227a9b549f84647\": container with ID starting with bd647846204296901aad949ad508b52a6702f05dfc78fcd04227a9b549f84647 not found: ID does not exist" containerID="bd647846204296901aad949ad508b52a6702f05dfc78fcd04227a9b549f84647" Jul 15 12:32:58.697141 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.697083 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd647846204296901aad949ad508b52a6702f05dfc78fcd04227a9b549f84647"} err="failed to get container status \"bd647846204296901aad949ad508b52a6702f05dfc78fcd04227a9b549f84647\": rpc error: code = NotFound desc = could not find container \"bd647846204296901aad949ad508b52a6702f05dfc78fcd04227a9b549f84647\": container with ID starting with bd647846204296901aad949ad508b52a6702f05dfc78fcd04227a9b549f84647 not found: ID does not exist" Jul 15 12:32:58.708240 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.708204 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f"] Jul 15 12:32:58.713202 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:58.713168 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d3e5d67b-dc56675d7-rxz6f"] Jul 15 12:32:59.092292 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:32:59.092251 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4c57401-6126-49a1-bba7-ce344ac1014f" path="/var/lib/kubelet/pods/b4c57401-6126-49a1-bba7-ce344ac1014f/volumes" Jul 15 12:33:05.013566 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:05.013529 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:08.960417 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:08.960372 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:09.024137 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:09.024095 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:13.505405 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:13.505369 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc"] Jul 15 12:33:13.505909 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:13.505624 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" podUID="2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" containerName="istio-proxy" containerID="cri-o://1cd23cfa7e4153e4bbf1fbac9704e421fe3a18281f845ac67edca8ac1b4b6a24" gracePeriod=30 Jul 15 12:33:13.528582 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:13.528543 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:13.769528 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:13.769406 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:13.897015 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:13.896960 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:13.991878 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:13.991836 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:14.154439 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:14.154409 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:18.761214 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.761182 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:33:18.795710 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.795598 2570 generic.go:356] "Generic (PLEG): container finished" podID="2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" containerID="1cd23cfa7e4153e4bbf1fbac9704e421fe3a18281f845ac67edca8ac1b4b6a24" exitCode=0 Jul 15 12:33:18.795710 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.795649 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" Jul 15 12:33:18.795710 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.795675 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" event={"ID":"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3","Type":"ContainerDied","Data":"1cd23cfa7e4153e4bbf1fbac9704e421fe3a18281f845ac67edca8ac1b4b6a24"} Jul 15 12:33:18.796004 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.795751 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc" event={"ID":"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3","Type":"ContainerDied","Data":"cf1bbd6a6b0f91f517eb7dd9539f5cc858342061521ee80613b3230f65260f66"} Jul 15 12:33:18.796004 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.795782 2570 scope.go:117] "RemoveContainer" containerID="1cd23cfa7e4153e4bbf1fbac9704e421fe3a18281f845ac67edca8ac1b4b6a24" Jul 15 12:33:18.806563 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.806543 2570 scope.go:117] "RemoveContainer" containerID="0867399d7431a52ce29ee66d91f1919af7e1f584440e16db2dd984521158727c" Jul 15 12:33:18.814789 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.814755 2570 scope.go:117] "RemoveContainer" containerID="1cd23cfa7e4153e4bbf1fbac9704e421fe3a18281f845ac67edca8ac1b4b6a24" Jul 15 12:33:18.815059 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:33:18.815038 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd23cfa7e4153e4bbf1fbac9704e421fe3a18281f845ac67edca8ac1b4b6a24\": container with ID starting with 1cd23cfa7e4153e4bbf1fbac9704e421fe3a18281f845ac67edca8ac1b4b6a24 not found: ID does not exist" containerID="1cd23cfa7e4153e4bbf1fbac9704e421fe3a18281f845ac67edca8ac1b4b6a24" Jul 15 12:33:18.815107 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.815067 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd23cfa7e4153e4bbf1fbac9704e421fe3a18281f845ac67edca8ac1b4b6a24"} err="failed to get container status \"1cd23cfa7e4153e4bbf1fbac9704e421fe3a18281f845ac67edca8ac1b4b6a24\": rpc error: code = NotFound desc = could not find container \"1cd23cfa7e4153e4bbf1fbac9704e421fe3a18281f845ac67edca8ac1b4b6a24\": container with ID starting with 1cd23cfa7e4153e4bbf1fbac9704e421fe3a18281f845ac67edca8ac1b4b6a24 not found: ID does not exist" Jul 15 12:33:18.815107 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.815088 2570 scope.go:117] "RemoveContainer" containerID="0867399d7431a52ce29ee66d91f1919af7e1f584440e16db2dd984521158727c" Jul 15 12:33:18.815354 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:33:18.815334 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0867399d7431a52ce29ee66d91f1919af7e1f584440e16db2dd984521158727c\": container with ID starting with 0867399d7431a52ce29ee66d91f1919af7e1f584440e16db2dd984521158727c not found: ID does not exist" containerID="0867399d7431a52ce29ee66d91f1919af7e1f584440e16db2dd984521158727c" Jul 15 12:33:18.815449 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.815361 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0867399d7431a52ce29ee66d91f1919af7e1f584440e16db2dd984521158727c"} err="failed to get container status \"0867399d7431a52ce29ee66d91f1919af7e1f584440e16db2dd984521158727c\": rpc error: code = NotFound desc = could not find container \"0867399d7431a52ce29ee66d91f1919af7e1f584440e16db2dd984521158727c\": container with ID starting with 0867399d7431a52ce29ee66d91f1919af7e1f584440e16db2dd984521158727c not found: ID does not exist" Jul 15 12:33:18.861990 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.861951 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-data\") pod \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " Jul 15 12:33:18.862169 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.862006 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-token\") pod \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " Jul 15 12:33:18.862169 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.862036 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-workload-socket\") pod \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " Jul 15 12:33:18.862169 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.862066 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istiod-ca-cert\") pod \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " Jul 15 12:33:18.862169 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.862091 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-podinfo\") pod \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " Jul 15 12:33:18.862169 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.862116 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-credential-socket\") pod \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " Jul 15 12:33:18.862169 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.862142 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-workload-certs\") pod \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " Jul 15 12:33:18.862470 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.862173 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-envoy\") pod \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " Jul 15 12:33:18.862470 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.862205 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lccd\" (UniqueName: \"kubernetes.io/projected/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-kube-api-access-2lccd\") pod \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\" (UID: \"2f96e492-69ef-4bd0-bd29-f59e93b4cdb3\") " Jul 15 12:33:18.862470 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.862304 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" (UID: "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:33:18.862618 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.862502 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" (UID: "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:33:18.862618 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.862505 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" (UID: "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:33:18.862618 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.862573 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-workload-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:33:18.862797 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.862698 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" (UID: "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:33:18.863022 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.862962 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-data" (OuterVolumeSpecName: "istio-data") pod "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" (UID: "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:33:18.864414 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.864385 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-token" (OuterVolumeSpecName: "istio-token") pod "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" (UID: "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:33:18.864556 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.864465 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-kube-api-access-2lccd" (OuterVolumeSpecName: "kube-api-access-2lccd") pod "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" (UID: "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3"). InnerVolumeSpecName "kube-api-access-2lccd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:33:18.864556 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.864526 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" (UID: "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:33:18.864933 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.864914 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" (UID: "2f96e492-69ef-4bd0-bd29-f59e93b4cdb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:33:18.963828 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.963788 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-data\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:33:18.963828 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.963818 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-token\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:33:18.963828 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.963827 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istiod-ca-cert\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:33:18.964067 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.963836 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-podinfo\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:33:18.964067 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.963850 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-credential-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:33:18.964067 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.963860 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-workload-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:33:18.964067 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.963869 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-istio-envoy\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:33:18.964067 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:18.963877 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lccd\" (UniqueName: \"kubernetes.io/projected/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3-kube-api-access-2lccd\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:33:19.115066 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:19.115036 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc"] Jul 15 12:33:19.121297 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:19.121269 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d3e5d67b-openshift-default-7bf689c75-4f5cc"] Jul 15 12:33:20.292461 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:20.292428 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:20.398730 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:20.398693 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:20.494095 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:20.494053 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:20.637975 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:20.637938 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:21.091964 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:21.091868 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" path="/var/lib/kubelet/pods/2f96e492-69ef-4bd0-bd29-f59e93b4cdb3/volumes" Jul 15 12:33:26.568018 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:26.567979 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:26.669577 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:26.669539 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:33.476951 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.476909 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:33.781876 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.781795 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf"] Jul 15 12:33:33.782261 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.782245 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4c57401-6126-49a1-bba7-ce344ac1014f" containerName="maas-api" Jul 15 12:33:33.782352 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.782267 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4c57401-6126-49a1-bba7-ce344ac1014f" containerName="maas-api" Jul 15 12:33:33.782352 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.782280 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" containerName="istio-proxy" Jul 15 12:33:33.782352 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.782286 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" containerName="istio-proxy" Jul 15 12:33:33.782352 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.782311 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28edefef-bbd5-4399-91e3-9f917d2bda52" containerName="revoke-keys" Jul 15 12:33:33.782352 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.782320 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="28edefef-bbd5-4399-91e3-9f917d2bda52" containerName="revoke-keys" Jul 15 12:33:33.782524 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.782379 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" containerName="istio-proxy" Jul 15 12:33:33.782524 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.782389 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4c57401-6126-49a1-bba7-ce344ac1014f" containerName="maas-api" Jul 15 12:33:33.782524 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.782394 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="28edefef-bbd5-4399-91e3-9f917d2bda52" containerName="revoke-keys" Jul 15 12:33:33.782524 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.782401 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" containerName="istio-proxy" Jul 15 12:33:33.784713 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.784695 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.786874 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.786849 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 12:33:33.787624 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.787605 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-145feecc-openshift-default-dockercfg-nvsmc\"" Jul 15 12:33:33.797258 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.797224 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf"] Jul 15 12:33:33.808150 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.808120 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:33.896208 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.896174 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/170351e8-22ef-4492-8920-2ee2122329b1-istio-token\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.896208 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.896206 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-istio-envoy\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.896443 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.896254 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-istio-data\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.896443 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.896282 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-workload-certs\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.896443 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.896308 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-workload-socket\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.896443 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.896367 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slq8g\" (UniqueName: \"kubernetes.io/projected/170351e8-22ef-4492-8920-2ee2122329b1-kube-api-access-slq8g\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.896443 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.896391 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-credential-socket\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.896443 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.896411 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/170351e8-22ef-4492-8920-2ee2122329b1-istiod-ca-cert\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.896443 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.896431 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/170351e8-22ef-4492-8920-2ee2122329b1-istio-podinfo\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.996930 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.996894 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-istio-data\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.997160 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.996944 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-workload-certs\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.997160 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.996984 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-workload-socket\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.997160 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.997045 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slq8g\" (UniqueName: \"kubernetes.io/projected/170351e8-22ef-4492-8920-2ee2122329b1-kube-api-access-slq8g\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.997160 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.997075 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-credential-socket\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.997160 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.997097 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/170351e8-22ef-4492-8920-2ee2122329b1-istiod-ca-cert\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.997160 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.997121 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/170351e8-22ef-4492-8920-2ee2122329b1-istio-podinfo\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.997455 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.997176 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/170351e8-22ef-4492-8920-2ee2122329b1-istio-token\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.997455 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.997204 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-istio-envoy\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.997455 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.997435 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-istio-data\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.997651 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.997629 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-workload-certs\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.997732 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.997664 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-credential-socket\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.997804 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.997763 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-workload-socket\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.998117 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.998092 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/170351e8-22ef-4492-8920-2ee2122329b1-istiod-ca-cert\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:33.999760 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.999720 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-istio-envoy\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:34.000015 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:33.999997 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/170351e8-22ef-4492-8920-2ee2122329b1-istio-podinfo\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:34.021436 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:34.021408 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/170351e8-22ef-4492-8920-2ee2122329b1-istio-token\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:34.021778 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:34.021754 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slq8g\" (UniqueName: \"kubernetes.io/projected/170351e8-22ef-4492-8920-2ee2122329b1-kube-api-access-slq8g\") pod \"e2e-mt-145feecc-openshift-default-7c85799-ttvnf\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:34.095640 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:34.095610 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:34.228287 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:34.228234 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:34.260684 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:34.260649 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf"] Jul 15 12:33:34.263870 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:33:34.263843 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170351e8_22ef_4492_8920_2ee2122329b1.slice/crio-71316bc800bd20b6b778adb4b2f0311a0e127be999e60922fedd3d9d0e3e9711 WatchSource:0}: Error finding container 71316bc800bd20b6b778adb4b2f0311a0e127be999e60922fedd3d9d0e3e9711: Status 404 returned error can't find the container with id 71316bc800bd20b6b778adb4b2f0311a0e127be999e60922fedd3d9d0e3e9711 Jul 15 12:33:34.266156 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:34.266123 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:33:34.266250 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:34.266193 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:33:34.266250 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:34.266232 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:33:34.880639 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:34.880600 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" event={"ID":"170351e8-22ef-4492-8920-2ee2122329b1","Type":"ContainerStarted","Data":"a1fd4f0fbe648ddf606b40436f9945ac9e1ea6cbd83684f04cdad75dc6dc55ad"} Jul 15 12:33:34.881074 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:34.880654 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" event={"ID":"170351e8-22ef-4492-8920-2ee2122329b1","Type":"ContainerStarted","Data":"71316bc800bd20b6b778adb4b2f0311a0e127be999e60922fedd3d9d0e3e9711"} Jul 15 12:33:34.916148 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:34.916096 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" podStartSLOduration=1.916080483 podStartE2EDuration="1.916080483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:33:34.913921894 +0000 UTC m=+2082.327450246" watchObservedRunningTime="2026-07-15 12:33:34.916080483 +0000 UTC m=+2082.329608831" Jul 15 12:33:35.096895 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:35.096858 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:35.101820 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:35.101795 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:35.886451 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:35.886418 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:35.887465 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:35.887444 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:36.822317 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:36.822277 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:36.921660 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:36.921623 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:37.045373 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.045336 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv"] Jul 15 12:33:37.045734 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.045721 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" containerName="istio-proxy" Jul 15 12:33:37.045734 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.045750 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f96e492-69ef-4bd0-bd29-f59e93b4cdb3" containerName="istio-proxy" Jul 15 12:33:37.047940 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.047923 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" Jul 15 12:33:37.050106 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.050083 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-145feecc-serving-cert\"" Jul 15 12:33:37.059550 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.059529 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv"] Jul 15 12:33:37.128038 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.127999 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77510266-57ff-4ed0-9f9c-febf0d140fd3-openshift-service-ca\") pod \"maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv\" (UID: \"77510266-57ff-4ed0-9f9c-febf0d140fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" Jul 15 12:33:37.128203 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.128048 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77510266-57ff-4ed0-9f9c-febf0d140fd3-maas-api-tls\") pod \"maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv\" (UID: \"77510266-57ff-4ed0-9f9c-febf0d140fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" Jul 15 12:33:37.128203 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.128138 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9rqb\" (UniqueName: \"kubernetes.io/projected/77510266-57ff-4ed0-9f9c-febf0d140fd3-kube-api-access-q9rqb\") pod \"maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv\" (UID: \"77510266-57ff-4ed0-9f9c-febf0d140fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" Jul 15 12:33:37.133201 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.133172 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:37.229575 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.229505 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77510266-57ff-4ed0-9f9c-febf0d140fd3-openshift-service-ca\") pod \"maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv\" (UID: \"77510266-57ff-4ed0-9f9c-febf0d140fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" Jul 15 12:33:37.229817 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.229588 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77510266-57ff-4ed0-9f9c-febf0d140fd3-maas-api-tls\") pod \"maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv\" (UID: \"77510266-57ff-4ed0-9f9c-febf0d140fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" Jul 15 12:33:37.229817 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.229642 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9rqb\" (UniqueName: \"kubernetes.io/projected/77510266-57ff-4ed0-9f9c-febf0d140fd3-kube-api-access-q9rqb\") pod \"maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv\" (UID: \"77510266-57ff-4ed0-9f9c-febf0d140fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" Jul 15 12:33:37.230271 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.230245 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77510266-57ff-4ed0-9f9c-febf0d140fd3-openshift-service-ca\") pod \"maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv\" (UID: \"77510266-57ff-4ed0-9f9c-febf0d140fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" Jul 15 12:33:37.232157 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.232129 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77510266-57ff-4ed0-9f9c-febf0d140fd3-maas-api-tls\") pod \"maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv\" (UID: \"77510266-57ff-4ed0-9f9c-febf0d140fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" Jul 15 12:33:37.232526 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.232505 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:37.237529 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.237511 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9rqb\" (UniqueName: \"kubernetes.io/projected/77510266-57ff-4ed0-9f9c-febf0d140fd3-kube-api-access-q9rqb\") pod \"maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv\" (UID: \"77510266-57ff-4ed0-9f9c-febf0d140fd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" Jul 15 12:33:37.359073 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.359037 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" Jul 15 12:33:37.501882 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.501857 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv"] Jul 15 12:33:37.504191 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:33:37.504163 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77510266_57ff_4ed0_9f9c_febf0d140fd3.slice/crio-51e0a0b57c25c38ddba4ee0e5a3951382fc0ccfcfcc020dd90eb456e344701cd WatchSource:0}: Error finding container 51e0a0b57c25c38ddba4ee0e5a3951382fc0ccfcfcc020dd90eb456e344701cd: Status 404 returned error can't find the container with id 51e0a0b57c25c38ddba4ee0e5a3951382fc0ccfcfcc020dd90eb456e344701cd Jul 15 12:33:37.621568 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.621533 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:37.724564 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.724466 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:37.897284 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:37.897246 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" event={"ID":"77510266-57ff-4ed0-9f9c-febf0d140fd3","Type":"ContainerStarted","Data":"51e0a0b57c25c38ddba4ee0e5a3951382fc0ccfcfcc020dd90eb456e344701cd"} Jul 15 12:33:38.905205 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:38.905171 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" event={"ID":"77510266-57ff-4ed0-9f9c-febf0d140fd3","Type":"ContainerStarted","Data":"87e79118bcfa20b437bdfb6073cf3687cd240b91b77881ec5eb4639481f99c95"} Jul 15 12:33:38.905567 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:38.905229 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" Jul 15 12:33:38.922953 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:38.922907 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" podStartSLOduration=1.523276434 podStartE2EDuration="1.922891329s" totalImagesPullingTime="399.614895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:33:38.921136186 +0000 UTC m=+2086.334664536" watchObservedRunningTime="2026-07-15 12:33:38.922891329 +0000 UTC m=+2086.336419741" Jul 15 12:33:41.363985 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:41.363949 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:44.916227 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:44.916196 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" Jul 15 12:33:50.294141 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:50.294103 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:50.395196 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:50.395158 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:51.490970 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:51.490530 2570 patch_prober.go:28] interesting pod/e2e-mt-145feecc-openshift-default-7c85799-ttvnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:42280->10.133.0.55:15021: read: connection reset by peer" start-of-body= Jul 15 12:33:51.490970 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:51.490627 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" podUID="170351e8-22ef-4492-8920-2ee2122329b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:42280->10.133.0.55:15021: read: connection reset by peer" Jul 15 12:33:51.976000 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:51.975962 2570 generic.go:356] "Generic (PLEG): container finished" podID="170351e8-22ef-4492-8920-2ee2122329b1" containerID="a1fd4f0fbe648ddf606b40436f9945ac9e1ea6cbd83684f04cdad75dc6dc55ad" exitCode=0 Jul 15 12:33:51.976000 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:51.976001 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" event={"ID":"170351e8-22ef-4492-8920-2ee2122329b1","Type":"ContainerDied","Data":"a1fd4f0fbe648ddf606b40436f9945ac9e1ea6cbd83684f04cdad75dc6dc55ad"} Jul 15 12:33:51.976394 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:51.976379 2570 scope.go:117] "RemoveContainer" containerID="a1fd4f0fbe648ddf606b40436f9945ac9e1ea6cbd83684f04cdad75dc6dc55ad" Jul 15 12:33:51.977405 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:51.977376 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:33:51.977472 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:51.977449 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:33:51.977513 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:51.977484 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:33:52.988812 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:52.988775 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" event={"ID":"170351e8-22ef-4492-8920-2ee2122329b1","Type":"ContainerStarted","Data":"4223727e2b1248c59b701e67dc8fa3eea96c943f0058c23df54b6a43b35ebaaf"} Jul 15 12:33:53.096454 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:53.096419 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:53.096700 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:53.096678 2570 patch_prober.go:28] interesting pod/e2e-mt-145feecc-openshift-default-7c85799-ttvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 15 12:33:53.096835 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:53.096730 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" podUID="170351e8-22ef-4492-8920-2ee2122329b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 15 12:33:53.259568 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:53.259420 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:33:53.290575 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:53.290546 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:33:54.096868 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:54.096833 2570 patch_prober.go:28] interesting pod/e2e-mt-145feecc-openshift-default-7c85799-ttvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 15 12:33:54.097269 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:54.096890 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" podUID="170351e8-22ef-4492-8920-2ee2122329b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 15 12:33:55.096048 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:55.096001 2570 patch_prober.go:28] interesting pod/e2e-mt-145feecc-openshift-default-7c85799-ttvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 15 12:33:55.096257 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:55.096067 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" podUID="170351e8-22ef-4492-8920-2ee2122329b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 15 12:33:56.096792 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:56.096720 2570 patch_prober.go:28] interesting pod/e2e-mt-145feecc-openshift-default-7c85799-ttvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 15 12:33:56.097214 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:56.096825 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" podUID="170351e8-22ef-4492-8920-2ee2122329b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 15 12:33:57.099306 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:57.099279 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:57.099767 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:57.099554 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:57.100246 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:57.100229 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:33:58.413932 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.413897 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:58.511896 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.511862 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:58.841572 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.841490 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96"] Jul 15 12:33:58.844093 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.844073 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:58.846367 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.846343 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-145feecc\"/\"openshift-service-ca.crt\"" Jul 15 12:33:58.847179 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.847159 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-145feecc\"/\"kube-root-ca.crt\"" Jul 15 12:33:58.847288 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.847192 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-145feecc\"/\"default-dockercfg-946cz\"" Jul 15 12:33:58.847288 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.847242 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-145feecc\"/\"e2e-lifecycle-model-145feecc-kserve-self-signed-certs\"" Jul 15 12:33:58.855954 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.855925 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96"] Jul 15 12:33:58.864000 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.863964 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:58.937836 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.937804 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-home\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:58.937836 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.937841 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-dshm\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:58.938111 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.937865 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-model-cache\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:58.938111 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.937988 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66550d8b-8a65-42f1-bd37-e3feafd04bcb-tls-certs\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:58.938111 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.938032 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcs7p\" (UniqueName: \"kubernetes.io/projected/66550d8b-8a65-42f1-bd37-e3feafd04bcb-kube-api-access-wcs7p\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:58.938274 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:58.938122 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-tmp-dir\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:59.039640 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:59.039600 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66550d8b-8a65-42f1-bd37-e3feafd04bcb-tls-certs\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:59.039640 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:59.039637 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcs7p\" (UniqueName: \"kubernetes.io/projected/66550d8b-8a65-42f1-bd37-e3feafd04bcb-kube-api-access-wcs7p\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:59.039914 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:59.039689 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-tmp-dir\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:59.039914 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:59.039728 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-home\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:59.039914 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:59.039771 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-dshm\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:59.039914 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:59.039796 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-model-cache\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:59.040217 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:59.040195 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-home\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:59.040304 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:59.040224 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-model-cache\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:59.040304 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:59.040275 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-tmp-dir\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:59.042231 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:59.042198 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-dshm\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:59.042525 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:59.042502 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66550d8b-8a65-42f1-bd37-e3feafd04bcb-tls-certs\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:59.047499 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:59.047478 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcs7p\" (UniqueName: \"kubernetes.io/projected/66550d8b-8a65-42f1-bd37-e3feafd04bcb-kube-api-access-wcs7p\") pod \"e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:59.153601 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:59.153556 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:33:59.288792 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:33:59.288763 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96"] Jul 15 12:33:59.290948 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:33:59.290922 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66550d8b_8a65_42f1_bd37_e3feafd04bcb.slice/crio-d4a63aeea8ce3960bf2bc26087b5cc5824bb9b3e1aca55dd90b02e47746aaae6 WatchSource:0}: Error finding container d4a63aeea8ce3960bf2bc26087b5cc5824bb9b3e1aca55dd90b02e47746aaae6: Status 404 returned error can't find the container with id d4a63aeea8ce3960bf2bc26087b5cc5824bb9b3e1aca55dd90b02e47746aaae6 Jul 15 12:34:00.025789 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:00.025723 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" event={"ID":"66550d8b-8a65-42f1-bd37-e3feafd04bcb","Type":"ContainerStarted","Data":"bc8b29ae34437b32c72570f9fcb0c1c982193eec0e15d621d8bac10081c08469"} Jul 15 12:34:00.025789 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:00.025796 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" event={"ID":"66550d8b-8a65-42f1-bd37-e3feafd04bcb","Type":"ContainerStarted","Data":"d4a63aeea8ce3960bf2bc26087b5cc5824bb9b3e1aca55dd90b02e47746aaae6"} Jul 15 12:34:00.044292 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:00.044239 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" podStartSLOduration=2.04422373 podStartE2EDuration="2.04422373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:34:00.04214892 +0000 UTC m=+2107.455677284" watchObservedRunningTime="2026-07-15 12:34:00.04422373 +0000 UTC m=+2107.457752082" Jul 15 12:34:09.153700 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:09.153668 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:34:09.154115 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:09.153715 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:34:09.166485 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:09.166459 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:34:10.088118 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:10.088092 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:34:27.996393 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:27.996355 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:30.261989 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:30.261955 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv"] Jul 15 12:34:30.262422 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:30.262221 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" podUID="77510266-57ff-4ed0-9f9c-febf0d140fd3" containerName="maas-api" containerID="cri-o://87e79118bcfa20b437bdfb6073cf3687cd240b91b77881ec5eb4639481f99c95" gracePeriod=30 Jul 15 12:34:30.292340 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:30.292310 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:30.508363 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:30.508334 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" Jul 15 12:34:30.632016 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:30.631973 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9rqb\" (UniqueName: \"kubernetes.io/projected/77510266-57ff-4ed0-9f9c-febf0d140fd3-kube-api-access-q9rqb\") pod \"77510266-57ff-4ed0-9f9c-febf0d140fd3\" (UID: \"77510266-57ff-4ed0-9f9c-febf0d140fd3\") " Jul 15 12:34:30.632203 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:30.632094 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77510266-57ff-4ed0-9f9c-febf0d140fd3-openshift-service-ca\") pod \"77510266-57ff-4ed0-9f9c-febf0d140fd3\" (UID: \"77510266-57ff-4ed0-9f9c-febf0d140fd3\") " Jul 15 12:34:30.632203 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:30.632123 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77510266-57ff-4ed0-9f9c-febf0d140fd3-maas-api-tls\") pod \"77510266-57ff-4ed0-9f9c-febf0d140fd3\" (UID: \"77510266-57ff-4ed0-9f9c-febf0d140fd3\") " Jul 15 12:34:30.632461 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:30.632439 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77510266-57ff-4ed0-9f9c-febf0d140fd3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77510266-57ff-4ed0-9f9c-febf0d140fd3" (UID: "77510266-57ff-4ed0-9f9c-febf0d140fd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:34:30.634301 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:30.634279 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77510266-57ff-4ed0-9f9c-febf0d140fd3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77510266-57ff-4ed0-9f9c-febf0d140fd3" (UID: "77510266-57ff-4ed0-9f9c-febf0d140fd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:34:30.634363 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:30.634335 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77510266-57ff-4ed0-9f9c-febf0d140fd3-kube-api-access-q9rqb" (OuterVolumeSpecName: "kube-api-access-q9rqb") pod "77510266-57ff-4ed0-9f9c-febf0d140fd3" (UID: "77510266-57ff-4ed0-9f9c-febf0d140fd3"). InnerVolumeSpecName "kube-api-access-q9rqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:34:30.733617 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:30.733580 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9rqb\" (UniqueName: \"kubernetes.io/projected/77510266-57ff-4ed0-9f9c-febf0d140fd3-kube-api-access-q9rqb\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:34:30.733617 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:30.733611 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77510266-57ff-4ed0-9f9c-febf0d140fd3-openshift-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:34:30.733617 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:30.733622 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77510266-57ff-4ed0-9f9c-febf0d140fd3-maas-api-tls\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:34:31.188997 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:31.188966 2570 generic.go:356] "Generic (PLEG): container finished" podID="77510266-57ff-4ed0-9f9c-febf0d140fd3" containerID="87e79118bcfa20b437bdfb6073cf3687cd240b91b77881ec5eb4639481f99c95" exitCode=0 Jul 15 12:34:31.189177 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:31.189013 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" Jul 15 12:34:31.189177 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:31.189052 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" event={"ID":"77510266-57ff-4ed0-9f9c-febf0d140fd3","Type":"ContainerDied","Data":"87e79118bcfa20b437bdfb6073cf3687cd240b91b77881ec5eb4639481f99c95"} Jul 15 12:34:31.189177 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:31.189111 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv" event={"ID":"77510266-57ff-4ed0-9f9c-febf0d140fd3","Type":"ContainerDied","Data":"51e0a0b57c25c38ddba4ee0e5a3951382fc0ccfcfcc020dd90eb456e344701cd"} Jul 15 12:34:31.189177 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:31.189134 2570 scope.go:117] "RemoveContainer" containerID="87e79118bcfa20b437bdfb6073cf3687cd240b91b77881ec5eb4639481f99c95" Jul 15 12:34:31.197598 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:31.197575 2570 scope.go:117] "RemoveContainer" containerID="87e79118bcfa20b437bdfb6073cf3687cd240b91b77881ec5eb4639481f99c95" Jul 15 12:34:31.197888 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:34:31.197862 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e79118bcfa20b437bdfb6073cf3687cd240b91b77881ec5eb4639481f99c95\": container with ID starting with 87e79118bcfa20b437bdfb6073cf3687cd240b91b77881ec5eb4639481f99c95 not found: ID does not exist" containerID="87e79118bcfa20b437bdfb6073cf3687cd240b91b77881ec5eb4639481f99c95" Jul 15 12:34:31.197989 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:31.197896 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e79118bcfa20b437bdfb6073cf3687cd240b91b77881ec5eb4639481f99c95"} err="failed to get container status \"87e79118bcfa20b437bdfb6073cf3687cd240b91b77881ec5eb4639481f99c95\": rpc error: code = NotFound desc = could not find container \"87e79118bcfa20b437bdfb6073cf3687cd240b91b77881ec5eb4639481f99c95\": container with ID starting with 87e79118bcfa20b437bdfb6073cf3687cd240b91b77881ec5eb4639481f99c95 not found: ID does not exist" Jul 15 12:34:31.226401 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:31.226373 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv"] Jul 15 12:34:31.235314 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:31.235293 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-145feecc-64dcc4b5df-h24pv"] Jul 15 12:34:33.092014 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:33.091974 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77510266-57ff-4ed0-9f9c-febf0d140fd3" path="/var/lib/kubelet/pods/77510266-57ff-4ed0-9f9c-febf0d140fd3/volumes" Jul 15 12:34:34.885851 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:34.885812 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:36.320574 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:36.320536 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:39.870203 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:39.870164 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96"] Jul 15 12:34:39.916412 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:39.916364 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:53.349667 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:53.349587 2570 scope.go:117] "RemoveContainer" containerID="96830b73b70abc9c9720a50f579c2be6e78940778889a44b224b5ea470e0fc75" Jul 15 12:34:54.891217 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:54.891166 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" podUID="66550d8b-8a65-42f1-bd37-e3feafd04bcb" containerName="main" containerID="cri-o://bc8b29ae34437b32c72570f9fcb0c1c982193eec0e15d621d8bac10081c08469" gracePeriod=45 Jul 15 12:34:55.141265 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.141203 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:34:55.153702 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.153676 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-model-cache\") pod \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " Jul 15 12:34:55.153854 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.153716 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcs7p\" (UniqueName: \"kubernetes.io/projected/66550d8b-8a65-42f1-bd37-e3feafd04bcb-kube-api-access-wcs7p\") pod \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " Jul 15 12:34:55.153854 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.153758 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66550d8b-8a65-42f1-bd37-e3feafd04bcb-tls-certs\") pod \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " Jul 15 12:34:55.153854 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.153780 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-tmp-dir\") pod \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " Jul 15 12:34:55.153854 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.153851 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-dshm\") pod \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " Jul 15 12:34:55.154063 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.153951 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-home\") pod \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\" (UID: \"66550d8b-8a65-42f1-bd37-e3feafd04bcb\") " Jul 15 12:34:55.154063 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.153976 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "66550d8b-8a65-42f1-bd37-e3feafd04bcb" (UID: "66550d8b-8a65-42f1-bd37-e3feafd04bcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:34:55.154167 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.154077 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66550d8b-8a65-42f1-bd37-e3feafd04bcb" (UID: "66550d8b-8a65-42f1-bd37-e3feafd04bcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:34:55.154264 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.154245 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-home" (OuterVolumeSpecName: "home") pod "66550d8b-8a65-42f1-bd37-e3feafd04bcb" (UID: "66550d8b-8a65-42f1-bd37-e3feafd04bcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:34:55.154340 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.154321 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-model-cache\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:34:55.154397 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.154345 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-tmp-dir\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:34:55.154397 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.154359 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-home\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:34:55.156146 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.156127 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66550d8b-8a65-42f1-bd37-e3feafd04bcb-kube-api-access-wcs7p" (OuterVolumeSpecName: "kube-api-access-wcs7p") pod "66550d8b-8a65-42f1-bd37-e3feafd04bcb" (UID: "66550d8b-8a65-42f1-bd37-e3feafd04bcb"). InnerVolumeSpecName "kube-api-access-wcs7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:34:55.156561 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.156540 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-dshm" (OuterVolumeSpecName: "dshm") pod "66550d8b-8a65-42f1-bd37-e3feafd04bcb" (UID: "66550d8b-8a65-42f1-bd37-e3feafd04bcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:34:55.156630 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.156557 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66550d8b-8a65-42f1-bd37-e3feafd04bcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66550d8b-8a65-42f1-bd37-e3feafd04bcb" (UID: "66550d8b-8a65-42f1-bd37-e3feafd04bcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:34:55.255098 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.255066 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcs7p\" (UniqueName: \"kubernetes.io/projected/66550d8b-8a65-42f1-bd37-e3feafd04bcb-kube-api-access-wcs7p\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:34:55.255098 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.255096 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66550d8b-8a65-42f1-bd37-e3feafd04bcb-tls-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:34:55.255098 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.255108 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66550d8b-8a65-42f1-bd37-e3feafd04bcb-dshm\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:34:55.326325 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.326290 2570 generic.go:356] "Generic (PLEG): container finished" podID="66550d8b-8a65-42f1-bd37-e3feafd04bcb" containerID="bc8b29ae34437b32c72570f9fcb0c1c982193eec0e15d621d8bac10081c08469" exitCode=0 Jul 15 12:34:55.326496 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.326333 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" Jul 15 12:34:55.326496 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.326338 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" event={"ID":"66550d8b-8a65-42f1-bd37-e3feafd04bcb","Type":"ContainerDied","Data":"bc8b29ae34437b32c72570f9fcb0c1c982193eec0e15d621d8bac10081c08469"} Jul 15 12:34:55.326496 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.326457 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96" event={"ID":"66550d8b-8a65-42f1-bd37-e3feafd04bcb","Type":"ContainerDied","Data":"d4a63aeea8ce3960bf2bc26087b5cc5824bb9b3e1aca55dd90b02e47746aaae6"} Jul 15 12:34:55.326496 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.326479 2570 scope.go:117] "RemoveContainer" containerID="bc8b29ae34437b32c72570f9fcb0c1c982193eec0e15d621d8bac10081c08469" Jul 15 12:34:55.336174 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.336157 2570 scope.go:117] "RemoveContainer" containerID="bc8b29ae34437b32c72570f9fcb0c1c982193eec0e15d621d8bac10081c08469" Jul 15 12:34:55.336429 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:34:55.336407 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc8b29ae34437b32c72570f9fcb0c1c982193eec0e15d621d8bac10081c08469\": container with ID starting with bc8b29ae34437b32c72570f9fcb0c1c982193eec0e15d621d8bac10081c08469 not found: ID does not exist" containerID="bc8b29ae34437b32c72570f9fcb0c1c982193eec0e15d621d8bac10081c08469" Jul 15 12:34:55.336475 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.336439 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc8b29ae34437b32c72570f9fcb0c1c982193eec0e15d621d8bac10081c08469"} err="failed to get container status \"bc8b29ae34437b32c72570f9fcb0c1c982193eec0e15d621d8bac10081c08469\": rpc error: code = NotFound desc = could not find container \"bc8b29ae34437b32c72570f9fcb0c1c982193eec0e15d621d8bac10081c08469\": container with ID starting with bc8b29ae34437b32c72570f9fcb0c1c982193eec0e15d621d8bac10081c08469 not found: ID does not exist" Jul 15 12:34:55.348264 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.348234 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96"] Jul 15 12:34:55.354106 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:55.354078 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-145feecc/e2e-lifecycle-model-145feecc-kserve-968788f5f-q6h96"] Jul 15 12:34:57.091706 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:34:57.091669 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66550d8b-8a65-42f1-bd37-e3feafd04bcb" path="/var/lib/kubelet/pods/66550d8b-8a65-42f1-bd37-e3feafd04bcb/volumes" Jul 15 12:35:16.200435 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:16.200395 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:16.850114 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:16.850083 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:22.812716 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:22.812670 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:22.875552 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:22.875519 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:27.174352 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:27.174320 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf"] Jul 15 12:35:27.174731 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:27.174546 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" podUID="170351e8-22ef-4492-8920-2ee2122329b1" containerName="istio-proxy" containerID="cri-o://4223727e2b1248c59b701e67dc8fa3eea96c943f0058c23df54b6a43b35ebaaf" gracePeriod=30 Jul 15 12:35:27.195030 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:27.195002 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:27.436478 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:27.436388 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:28.494317 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:28.494286 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:28.579132 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:28.579092 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:28.677915 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:28.677884 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:29.434541 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:29.434504 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:29.549502 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:29.549469 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:30.993864 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:30.993827 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:32.418358 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.418330 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:35:32.501289 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.501209 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slq8g\" (UniqueName: \"kubernetes.io/projected/170351e8-22ef-4492-8920-2ee2122329b1-kube-api-access-slq8g\") pod \"170351e8-22ef-4492-8920-2ee2122329b1\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " Jul 15 12:35:32.501289 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.501255 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/170351e8-22ef-4492-8920-2ee2122329b1-istio-podinfo\") pod \"170351e8-22ef-4492-8920-2ee2122329b1\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " Jul 15 12:35:32.501289 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.501279 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-workload-certs\") pod \"170351e8-22ef-4492-8920-2ee2122329b1\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " Jul 15 12:35:32.501562 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.501322 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-workload-socket\") pod \"170351e8-22ef-4492-8920-2ee2122329b1\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " Jul 15 12:35:32.501562 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.501348 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/170351e8-22ef-4492-8920-2ee2122329b1-istio-token\") pod \"170351e8-22ef-4492-8920-2ee2122329b1\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " Jul 15 12:35:32.501562 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.501369 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-istio-data\") pod \"170351e8-22ef-4492-8920-2ee2122329b1\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " Jul 15 12:35:32.501562 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.501398 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-istio-envoy\") pod \"170351e8-22ef-4492-8920-2ee2122329b1\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " Jul 15 12:35:32.501562 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.501472 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/170351e8-22ef-4492-8920-2ee2122329b1-istiod-ca-cert\") pod \"170351e8-22ef-4492-8920-2ee2122329b1\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " Jul 15 12:35:32.501562 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.501526 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-credential-socket\") pod \"170351e8-22ef-4492-8920-2ee2122329b1\" (UID: \"170351e8-22ef-4492-8920-2ee2122329b1\") " Jul 15 12:35:32.501972 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.501940 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "170351e8-22ef-4492-8920-2ee2122329b1" (UID: "170351e8-22ef-4492-8920-2ee2122329b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:35:32.502325 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.502195 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "170351e8-22ef-4492-8920-2ee2122329b1" (UID: "170351e8-22ef-4492-8920-2ee2122329b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:35:32.502452 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.502395 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170351e8-22ef-4492-8920-2ee2122329b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "170351e8-22ef-4492-8920-2ee2122329b1" (UID: "170351e8-22ef-4492-8920-2ee2122329b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:35:32.502866 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.502831 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "170351e8-22ef-4492-8920-2ee2122329b1" (UID: "170351e8-22ef-4492-8920-2ee2122329b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:35:32.502973 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.502926 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "170351e8-22ef-4492-8920-2ee2122329b1" (UID: "170351e8-22ef-4492-8920-2ee2122329b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:35:32.503908 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.503810 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "170351e8-22ef-4492-8920-2ee2122329b1" (UID: "170351e8-22ef-4492-8920-2ee2122329b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:35:32.504149 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.504105 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170351e8-22ef-4492-8920-2ee2122329b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "170351e8-22ef-4492-8920-2ee2122329b1" (UID: "170351e8-22ef-4492-8920-2ee2122329b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:35:32.504255 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.504200 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/170351e8-22ef-4492-8920-2ee2122329b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "170351e8-22ef-4492-8920-2ee2122329b1" (UID: "170351e8-22ef-4492-8920-2ee2122329b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:35:32.504317 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.504256 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170351e8-22ef-4492-8920-2ee2122329b1-kube-api-access-slq8g" (OuterVolumeSpecName: "kube-api-access-slq8g") pod "170351e8-22ef-4492-8920-2ee2122329b1" (UID: "170351e8-22ef-4492-8920-2ee2122329b1"). InnerVolumeSpecName "kube-api-access-slq8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:35:32.510579 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.510551 2570 generic.go:356] "Generic (PLEG): container finished" podID="170351e8-22ef-4492-8920-2ee2122329b1" containerID="4223727e2b1248c59b701e67dc8fa3eea96c943f0058c23df54b6a43b35ebaaf" exitCode=0 Jul 15 12:35:32.510678 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.510619 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" Jul 15 12:35:32.510678 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.510630 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" event={"ID":"170351e8-22ef-4492-8920-2ee2122329b1","Type":"ContainerDied","Data":"4223727e2b1248c59b701e67dc8fa3eea96c943f0058c23df54b6a43b35ebaaf"} Jul 15 12:35:32.510806 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.510703 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf" event={"ID":"170351e8-22ef-4492-8920-2ee2122329b1","Type":"ContainerDied","Data":"71316bc800bd20b6b778adb4b2f0311a0e127be999e60922fedd3d9d0e3e9711"} Jul 15 12:35:32.510806 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.510756 2570 scope.go:117] "RemoveContainer" containerID="4223727e2b1248c59b701e67dc8fa3eea96c943f0058c23df54b6a43b35ebaaf" Jul 15 12:35:32.527090 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.527066 2570 scope.go:117] "RemoveContainer" containerID="a1fd4f0fbe648ddf606b40436f9945ac9e1ea6cbd83684f04cdad75dc6dc55ad" Jul 15 12:35:32.536514 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.536492 2570 scope.go:117] "RemoveContainer" containerID="4223727e2b1248c59b701e67dc8fa3eea96c943f0058c23df54b6a43b35ebaaf" Jul 15 12:35:32.536841 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:35:32.536817 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4223727e2b1248c59b701e67dc8fa3eea96c943f0058c23df54b6a43b35ebaaf\": container with ID starting with 4223727e2b1248c59b701e67dc8fa3eea96c943f0058c23df54b6a43b35ebaaf not found: ID does not exist" containerID="4223727e2b1248c59b701e67dc8fa3eea96c943f0058c23df54b6a43b35ebaaf" Jul 15 12:35:32.536939 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.536853 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4223727e2b1248c59b701e67dc8fa3eea96c943f0058c23df54b6a43b35ebaaf"} err="failed to get container status \"4223727e2b1248c59b701e67dc8fa3eea96c943f0058c23df54b6a43b35ebaaf\": rpc error: code = NotFound desc = could not find container \"4223727e2b1248c59b701e67dc8fa3eea96c943f0058c23df54b6a43b35ebaaf\": container with ID starting with 4223727e2b1248c59b701e67dc8fa3eea96c943f0058c23df54b6a43b35ebaaf not found: ID does not exist" Jul 15 12:35:32.536939 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.536871 2570 scope.go:117] "RemoveContainer" containerID="a1fd4f0fbe648ddf606b40436f9945ac9e1ea6cbd83684f04cdad75dc6dc55ad" Jul 15 12:35:32.537153 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:35:32.537135 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fd4f0fbe648ddf606b40436f9945ac9e1ea6cbd83684f04cdad75dc6dc55ad\": container with ID starting with a1fd4f0fbe648ddf606b40436f9945ac9e1ea6cbd83684f04cdad75dc6dc55ad not found: ID does not exist" containerID="a1fd4f0fbe648ddf606b40436f9945ac9e1ea6cbd83684f04cdad75dc6dc55ad" Jul 15 12:35:32.537201 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.537160 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fd4f0fbe648ddf606b40436f9945ac9e1ea6cbd83684f04cdad75dc6dc55ad"} err="failed to get container status \"a1fd4f0fbe648ddf606b40436f9945ac9e1ea6cbd83684f04cdad75dc6dc55ad\": rpc error: code = NotFound desc = could not find container \"a1fd4f0fbe648ddf606b40436f9945ac9e1ea6cbd83684f04cdad75dc6dc55ad\": container with ID starting with a1fd4f0fbe648ddf606b40436f9945ac9e1ea6cbd83684f04cdad75dc6dc55ad not found: ID does not exist" Jul 15 12:35:32.537549 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.537532 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf"] Jul 15 12:35:32.543349 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.543328 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-145feecc-openshift-default-7c85799-ttvnf"] Jul 15 12:35:32.602395 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.602361 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/170351e8-22ef-4492-8920-2ee2122329b1-istiod-ca-cert\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:35:32.602395 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.602388 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-credential-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:35:32.602395 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.602402 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slq8g\" (UniqueName: \"kubernetes.io/projected/170351e8-22ef-4492-8920-2ee2122329b1-kube-api-access-slq8g\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:35:32.602636 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.602416 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/170351e8-22ef-4492-8920-2ee2122329b1-istio-podinfo\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:35:32.602636 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.602427 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-workload-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:35:32.602636 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.602440 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-workload-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:35:32.602636 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.602450 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/170351e8-22ef-4492-8920-2ee2122329b1-istio-token\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:35:32.602636 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.602462 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-istio-data\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:35:32.602636 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:32.602474 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/170351e8-22ef-4492-8920-2ee2122329b1-istio-envoy\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:35:33.097591 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:33.093660 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="170351e8-22ef-4492-8920-2ee2122329b1" path="/var/lib/kubelet/pods/170351e8-22ef-4492-8920-2ee2122329b1/volumes" Jul 15 12:35:36.901720 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:36.901669 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:37.002245 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:37.002206 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:37.594205 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:37.594171 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:43.487900 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:43.487861 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:45.045159 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:45.045107 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:54.222955 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:54.222923 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:54.582057 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:35:54.581973 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:15.552899 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:15.552807 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:22.211303 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.211269 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h"] Jul 15 12:36:22.211711 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.211688 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66550d8b-8a65-42f1-bd37-e3feafd04bcb" containerName="main" Jul 15 12:36:22.211711 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.211703 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="66550d8b-8a65-42f1-bd37-e3feafd04bcb" containerName="main" Jul 15 12:36:22.211850 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.211728 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="170351e8-22ef-4492-8920-2ee2122329b1" containerName="istio-proxy" Jul 15 12:36:22.211850 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.211752 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="170351e8-22ef-4492-8920-2ee2122329b1" containerName="istio-proxy" Jul 15 12:36:22.211850 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.211773 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="170351e8-22ef-4492-8920-2ee2122329b1" containerName="istio-proxy" Jul 15 12:36:22.211850 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.211780 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="170351e8-22ef-4492-8920-2ee2122329b1" containerName="istio-proxy" Jul 15 12:36:22.211850 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.211791 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77510266-57ff-4ed0-9f9c-febf0d140fd3" containerName="maas-api" Jul 15 12:36:22.211850 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.211796 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="77510266-57ff-4ed0-9f9c-febf0d140fd3" containerName="maas-api" Jul 15 12:36:22.212047 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.211863 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="170351e8-22ef-4492-8920-2ee2122329b1" containerName="istio-proxy" Jul 15 12:36:22.212047 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.211881 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="77510266-57ff-4ed0-9f9c-febf0d140fd3" containerName="maas-api" Jul 15 12:36:22.212047 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.211889 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="66550d8b-8a65-42f1-bd37-e3feafd04bcb" containerName="main" Jul 15 12:36:22.215051 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.215035 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.217038 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.216966 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 12:36:22.217038 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.216984 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-dbcba5-openshift-default-dockercfg-nmpw6\"" Jul 15 12:36:22.226132 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.226106 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h"] Jul 15 12:36:22.343014 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.342979 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwjg9\" (UniqueName: \"kubernetes.io/projected/13289992-9721-473f-8b4a-7237d7db78fe-kube-api-access-mwjg9\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.343216 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.343022 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-istio-envoy\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.343216 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.343096 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13289992-9721-473f-8b4a-7237d7db78fe-istio-token\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.343216 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.343131 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-istio-data\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.343216 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.343166 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-credential-socket\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.343414 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.343218 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-workload-socket\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.343414 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.343235 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13289992-9721-473f-8b4a-7237d7db78fe-istio-podinfo\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.343414 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.343257 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-workload-certs\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.343414 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.343272 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13289992-9721-473f-8b4a-7237d7db78fe-istiod-ca-cert\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.444156 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.444114 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwjg9\" (UniqueName: \"kubernetes.io/projected/13289992-9721-473f-8b4a-7237d7db78fe-kube-api-access-mwjg9\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.444358 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.444164 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-istio-envoy\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.444358 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.444209 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13289992-9721-473f-8b4a-7237d7db78fe-istio-token\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.444358 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.444251 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-istio-data\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.444358 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.444284 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-credential-socket\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.444560 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.444467 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-workload-socket\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.444560 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.444509 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13289992-9721-473f-8b4a-7237d7db78fe-istio-podinfo\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.444560 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.444553 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-workload-certs\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.444719 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.444573 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13289992-9721-473f-8b4a-7237d7db78fe-istiod-ca-cert\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.444719 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.444711 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-istio-data\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.444855 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.444767 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-credential-socket\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.444929 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.444909 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-workload-socket\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.445028 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.444997 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-workload-certs\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.445338 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.445322 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13289992-9721-473f-8b4a-7237d7db78fe-istiod-ca-cert\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.446662 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.446645 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-istio-envoy\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.446990 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.446971 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13289992-9721-473f-8b4a-7237d7db78fe-istio-podinfo\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.451336 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.451306 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13289992-9721-473f-8b4a-7237d7db78fe-istio-token\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.451438 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.451365 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwjg9\" (UniqueName: \"kubernetes.io/projected/13289992-9721-473f-8b4a-7237d7db78fe-kube-api-access-mwjg9\") pod \"e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.526149 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.526045 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:22.664100 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.664073 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h"] Jul 15 12:36:22.666282 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:36:22.666251 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13289992_9721_473f_8b4a_7237d7db78fe.slice/crio-3a9d19977d909dfcf50039844f8aee498f9ed3d424dae2e15adad628620e636f WatchSource:0}: Error finding container 3a9d19977d909dfcf50039844f8aee498f9ed3d424dae2e15adad628620e636f: Status 404 returned error can't find the container with id 3a9d19977d909dfcf50039844f8aee498f9ed3d424dae2e15adad628620e636f Jul 15 12:36:22.668450 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.668419 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:36:22.668551 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.668500 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:36:22.668614 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.668548 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:36:22.762352 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.762321 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" event={"ID":"13289992-9721-473f-8b4a-7237d7db78fe","Type":"ContainerStarted","Data":"6e8059c1d5aa9b9772ee7d237e75f4ac9b94b577d131e736ca1a97bc2d18f6f2"} Jul 15 12:36:22.762516 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.762372 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" event={"ID":"13289992-9721-473f-8b4a-7237d7db78fe","Type":"ContainerStarted","Data":"3a9d19977d909dfcf50039844f8aee498f9ed3d424dae2e15adad628620e636f"} Jul 15 12:36:22.783233 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:22.783140 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" podStartSLOduration=0.783124519 podStartE2EDuration="783.124519ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:36:22.780027849 +0000 UTC m=+2250.193556199" watchObservedRunningTime="2026-07-15 12:36:22.783124519 +0000 UTC m=+2250.196652870" Jul 15 12:36:23.526277 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:23.526242 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:23.531041 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:23.531012 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:23.767980 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:23.767943 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:23.768819 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:23.768799 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:24.055034 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:24.054998 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:24.155798 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:24.155765 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:24.806711 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:24.806679 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k"] Jul 15 12:36:24.807691 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:24.807659 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="170351e8-22ef-4492-8920-2ee2122329b1" containerName="istio-proxy" Jul 15 12:36:24.811178 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:24.811156 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:36:24.813244 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:24.813208 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-dbcba5-serving-cert\"" Jul 15 12:36:24.822477 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:24.822433 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k"] Jul 15 12:36:24.864429 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:24.864391 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1012c92-8790-4ce8-80be-94f12c2ecf81-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k\" (UID: \"b1012c92-8790-4ce8-80be-94f12c2ecf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:36:24.864633 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:24.864450 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1012c92-8790-4ce8-80be-94f12c2ecf81-maas-api-tls\") pod \"maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k\" (UID: \"b1012c92-8790-4ce8-80be-94f12c2ecf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:36:24.864633 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:24.864500 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l4vx\" (UniqueName: \"kubernetes.io/projected/b1012c92-8790-4ce8-80be-94f12c2ecf81-kube-api-access-9l4vx\") pod \"maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k\" (UID: \"b1012c92-8790-4ce8-80be-94f12c2ecf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:36:24.965915 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:24.965875 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1012c92-8790-4ce8-80be-94f12c2ecf81-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k\" (UID: \"b1012c92-8790-4ce8-80be-94f12c2ecf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:36:24.966131 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:24.965933 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1012c92-8790-4ce8-80be-94f12c2ecf81-maas-api-tls\") pod \"maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k\" (UID: \"b1012c92-8790-4ce8-80be-94f12c2ecf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:36:24.966131 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:24.965951 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l4vx\" (UniqueName: \"kubernetes.io/projected/b1012c92-8790-4ce8-80be-94f12c2ecf81-kube-api-access-9l4vx\") pod \"maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k\" (UID: \"b1012c92-8790-4ce8-80be-94f12c2ecf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:36:24.966131 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:36:24.966045 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-serving-cert: secret "maas-api-e2e-inf-a-dbcba5-serving-cert" not found Jul 15 12:36:24.966131 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:36:24.966123 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1012c92-8790-4ce8-80be-94f12c2ecf81-maas-api-tls podName:b1012c92-8790-4ce8-80be-94f12c2ecf81 nodeName:}" failed. No retries permitted until 2026-07-15 12:36:25.466102703 +0000 UTC m=+2252.879631033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1012c92-8790-4ce8-80be-94f12c2ecf81-maas-api-tls") pod "maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" (UID: "b1012c92-8790-4ce8-80be-94f12c2ecf81") : secret "maas-api-e2e-inf-a-dbcba5-serving-cert" not found Jul 15 12:36:24.966631 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:24.966607 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1012c92-8790-4ce8-80be-94f12c2ecf81-openshift-service-ca\") pod \"maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k\" (UID: \"b1012c92-8790-4ce8-80be-94f12c2ecf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:36:24.976288 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:24.976260 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l4vx\" (UniqueName: \"kubernetes.io/projected/b1012c92-8790-4ce8-80be-94f12c2ecf81-kube-api-access-9l4vx\") pod \"maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k\" (UID: \"b1012c92-8790-4ce8-80be-94f12c2ecf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:36:25.469323 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:25.469285 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1012c92-8790-4ce8-80be-94f12c2ecf81-maas-api-tls\") pod \"maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k\" (UID: \"b1012c92-8790-4ce8-80be-94f12c2ecf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:36:25.471825 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:25.471798 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1012c92-8790-4ce8-80be-94f12c2ecf81-maas-api-tls\") pod \"maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k\" (UID: \"b1012c92-8790-4ce8-80be-94f12c2ecf81\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:36:25.725817 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:25.725695 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:36:25.867346 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:25.867311 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k"] Jul 15 12:36:25.871120 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:36:25.871088 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1012c92_8790_4ce8_80be_94f12c2ecf81.slice/crio-b4b2d1621b179a217c8cba7ad37fe07a9366c6dad315b0b1f1ca8d1e55b5fbd7 WatchSource:0}: Error finding container b4b2d1621b179a217c8cba7ad37fe07a9366c6dad315b0b1f1ca8d1e55b5fbd7: Status 404 returned error can't find the container with id b4b2d1621b179a217c8cba7ad37fe07a9366c6dad315b0b1f1ca8d1e55b5fbd7 Jul 15 12:36:26.787468 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:26.787436 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" event={"ID":"b1012c92-8790-4ce8-80be-94f12c2ecf81","Type":"ContainerStarted","Data":"d39a0773e4c99e92cadcec79f3a42e681c07684a831d8991060dca28603ad4b7"} Jul 15 12:36:26.787660 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:26.787486 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" event={"ID":"b1012c92-8790-4ce8-80be-94f12c2ecf81","Type":"ContainerStarted","Data":"b4b2d1621b179a217c8cba7ad37fe07a9366c6dad315b0b1f1ca8d1e55b5fbd7"} Jul 15 12:36:26.787660 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:26.787528 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:36:26.805223 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:26.805178 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" podStartSLOduration=2.372579581 podStartE2EDuration="2.805163476s" totalImagesPullingTime="432.583895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:36:26.802903079 +0000 UTC m=+2254.216431429" watchObservedRunningTime="2026-07-15 12:36:26.805163476 +0000 UTC m=+2254.218691883" Jul 15 12:36:27.159640 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:27.159600 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:27.260659 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:27.260626 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:29.228945 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.228910 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:29.560200 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.560107 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:29.835757 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.835664 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm"] Jul 15 12:36:29.839347 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.839327 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:29.841241 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.841221 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6dba6e-openshift-default-dockercfg-5hgnt\"" Jul 15 12:36:29.852073 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.852048 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm"] Jul 15 12:36:29.859317 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.859288 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:29.910370 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.910334 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22668ef-408e-43a5-914f-953a544649eb-istio-podinfo\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:29.910370 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.910370 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-workload-socket\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:29.910642 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.910395 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-istio-data\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:29.910642 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.910493 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-istio-envoy\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:29.910642 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.910546 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgdm5\" (UniqueName: \"kubernetes.io/projected/f22668ef-408e-43a5-914f-953a544649eb-kube-api-access-jgdm5\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:29.910642 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.910589 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-workload-certs\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:29.910857 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.910670 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22668ef-408e-43a5-914f-953a544649eb-istiod-ca-cert\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:29.910857 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.910700 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-credential-socket\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:29.910857 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:29.910731 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22668ef-408e-43a5-914f-953a544649eb-istio-token\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.011397 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.011358 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22668ef-408e-43a5-914f-953a544649eb-istio-token\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.011397 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.011400 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22668ef-408e-43a5-914f-953a544649eb-istio-podinfo\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.011674 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.011418 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-workload-socket\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.011674 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.011447 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-istio-data\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.011674 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.011479 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-istio-envoy\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.011674 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.011510 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgdm5\" (UniqueName: \"kubernetes.io/projected/f22668ef-408e-43a5-914f-953a544649eb-kube-api-access-jgdm5\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.011674 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.011569 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-workload-certs\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.011989 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.011701 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22668ef-408e-43a5-914f-953a544649eb-istiod-ca-cert\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.011989 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.011730 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-credential-socket\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.011989 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.011903 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-workload-socket\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.011989 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.011948 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-istio-data\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.012196 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.012133 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-workload-certs\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.012196 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.012146 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-credential-socket\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.012427 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.012405 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22668ef-408e-43a5-914f-953a544649eb-istiod-ca-cert\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.014161 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.014135 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22668ef-408e-43a5-914f-953a544649eb-istio-podinfo\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.014352 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.014335 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-istio-envoy\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.019111 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.019085 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22668ef-408e-43a5-914f-953a544649eb-istio-token\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.019204 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.019147 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgdm5\" (UniqueName: \"kubernetes.io/projected/f22668ef-408e-43a5-914f-953a544649eb-kube-api-access-jgdm5\") pod \"e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.149474 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.149435 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:30.264785 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.264729 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:30.291526 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.291501 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm"] Jul 15 12:36:30.293527 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:36:30.293498 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22668ef_408e_43a5_914f_953a544649eb.slice/crio-7d905ba362fcb4102d8266a85b3ba95506d6827ee483bdf03e3ba743c65db53d WatchSource:0}: Error finding container 7d905ba362fcb4102d8266a85b3ba95506d6827ee483bdf03e3ba743c65db53d: Status 404 returned error can't find the container with id 7d905ba362fcb4102d8266a85b3ba95506d6827ee483bdf03e3ba743c65db53d Jul 15 12:36:30.295786 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.295726 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:36:30.295891 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.295817 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:36:30.295891 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.295844 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:36:30.810260 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.810221 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" event={"ID":"f22668ef-408e-43a5-914f-953a544649eb","Type":"ContainerStarted","Data":"3029a8c559aab996f3b2848bcbd3aa614e73f34b8bd8a6436cda7d3dbe19d6e7"} Jul 15 12:36:30.810446 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.810276 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" event={"ID":"f22668ef-408e-43a5-914f-953a544649eb","Type":"ContainerStarted","Data":"7d905ba362fcb4102d8266a85b3ba95506d6827ee483bdf03e3ba743c65db53d"} Jul 15 12:36:30.828713 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:30.828650 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" podStartSLOduration=1.828630944 podStartE2EDuration="1.828630944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:36:30.827392491 +0000 UTC m=+2258.240920842" watchObservedRunningTime="2026-07-15 12:36:30.828630944 +0000 UTC m=+2258.242159295" Jul 15 12:36:31.149830 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:31.149794 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:31.155032 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:31.155001 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:31.815884 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:31.815845 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:31.816860 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:31.816839 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:32.743680 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:32.743650 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:32.799331 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:32.799302 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:36:32.830043 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:32.830012 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:32.892526 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:32.892487 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:33.220152 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:33.220116 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:33.539675 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:33.539582 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:33.640575 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:33.640540 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:37.262535 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:37.262505 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:39.809979 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:39.809943 2570 patch_prober.go:28] interesting pod/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 12:36:39.810395 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:39.810030 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" podUID="13289992-9721-473f-8b4a-7237d7db78fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 12:36:40.895355 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:40.895319 2570 generic.go:356] "Generic (PLEG): container finished" podID="13289992-9721-473f-8b4a-7237d7db78fe" containerID="6e8059c1d5aa9b9772ee7d237e75f4ac9b94b577d131e736ca1a97bc2d18f6f2" exitCode=0 Jul 15 12:36:40.895933 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:40.895404 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" event={"ID":"13289992-9721-473f-8b4a-7237d7db78fe","Type":"ContainerDied","Data":"6e8059c1d5aa9b9772ee7d237e75f4ac9b94b577d131e736ca1a97bc2d18f6f2"} Jul 15 12:36:40.896044 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:40.896023 2570 scope.go:117] "RemoveContainer" containerID="6e8059c1d5aa9b9772ee7d237e75f4ac9b94b577d131e736ca1a97bc2d18f6f2" Jul 15 12:36:40.897252 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:40.897223 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:36:40.897377 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:40.897307 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:36:40.897377 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:40.897351 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:36:41.907027 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:41.906973 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" event={"ID":"13289992-9721-473f-8b4a-7237d7db78fe","Type":"ContainerStarted","Data":"f8c964563c0cbb63c0c804aa1dd11fe9f493c8573354bffff9ae5698d96284b8"} Jul 15 12:36:42.527188 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:42.527144 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:42.527405 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:42.527271 2570 patch_prober.go:28] interesting pod/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 15 12:36:42.527405 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:42.527322 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" podUID="13289992-9721-473f-8b4a-7237d7db78fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 15 12:36:43.527095 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:43.527050 2570 patch_prober.go:28] interesting pod/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 15 12:36:43.527519 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:43.527110 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" podUID="13289992-9721-473f-8b4a-7237d7db78fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 15 12:36:44.527329 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:44.527294 2570 patch_prober.go:28] interesting pod/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 15 12:36:44.527775 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:44.527355 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" podUID="13289992-9721-473f-8b4a-7237d7db78fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 15 12:36:45.527421 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:45.527385 2570 patch_prober.go:28] interesting pod/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 15 12:36:45.527931 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:45.527450 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" podUID="13289992-9721-473f-8b4a-7237d7db78fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 15 12:36:46.469570 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:46.469537 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:46.530404 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:46.530374 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:46.530893 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:46.530786 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:46.531489 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:46.531467 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:36:46.556478 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:46.556447 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:47.199634 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:47.198562 2570 patch_prober.go:28] interesting pod/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": read tcp 10.133.0.2:57048->10.133.0.60:15021: read: connection reset by peer" start-of-body= Jul 15 12:36:47.199634 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:47.198646 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" podUID="f22668ef-408e-43a5-914f-953a544649eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": read tcp 10.133.0.2:57048->10.133.0.60:15021: read: connection reset by peer" Jul 15 12:36:47.951559 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:47.951525 2570 generic.go:356] "Generic (PLEG): container finished" podID="f22668ef-408e-43a5-914f-953a544649eb" containerID="3029a8c559aab996f3b2848bcbd3aa614e73f34b8bd8a6436cda7d3dbe19d6e7" exitCode=0 Jul 15 12:36:47.952024 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:47.951598 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" event={"ID":"f22668ef-408e-43a5-914f-953a544649eb","Type":"ContainerDied","Data":"3029a8c559aab996f3b2848bcbd3aa614e73f34b8bd8a6436cda7d3dbe19d6e7"} Jul 15 12:36:47.952086 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:47.952068 2570 scope.go:117] "RemoveContainer" containerID="3029a8c559aab996f3b2848bcbd3aa614e73f34b8bd8a6436cda7d3dbe19d6e7" Jul 15 12:36:47.953086 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:47.953054 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:36:47.953145 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:47.953126 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:36:47.953181 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:47.953167 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:36:48.958155 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:48.958118 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" event={"ID":"f22668ef-408e-43a5-914f-953a544649eb","Type":"ContainerStarted","Data":"93563daf4f80cc6f2e48c23433df0453a96e20525b685270d17f39a76ea0db59"} Jul 15 12:36:49.150225 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:49.150186 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:49.150450 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:49.150428 2570 patch_prober.go:28] interesting pod/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 15 12:36:49.150508 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:49.150489 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" podUID="f22668ef-408e-43a5-914f-953a544649eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 15 12:36:50.150317 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.150281 2570 patch_prober.go:28] interesting pod/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 15 12:36:50.150711 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.150348 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" podUID="f22668ef-408e-43a5-914f-953a544649eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 15 12:36:50.243844 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.243811 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk"] Jul 15 12:36:50.247514 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.247499 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.249506 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.249480 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dbcba5\"/\"kube-root-ca.crt\"" Jul 15 12:36:50.249905 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.249887 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dbcba5\"/\"default-dockercfg-pll9t\"" Jul 15 12:36:50.250037 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.250019 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dbcba5\"/\"test-model-dbcba5-kserve-self-signed-certs\"" Jul 15 12:36:50.250127 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.250103 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dbcba5\"/\"openshift-service-ca.crt\"" Jul 15 12:36:50.256271 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.256248 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk"] Jul 15 12:36:50.315597 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.315555 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-model-cache\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.315812 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.315610 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-home\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.315812 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.315706 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-tmp-dir\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.315812 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.315767 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-dshm\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.315990 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.315826 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7kbj\" (UniqueName: \"kubernetes.io/projected/ba071e45-db2d-4d13-8d3e-ca0dd323d337-kube-api-access-z7kbj\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.315990 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.315946 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba071e45-db2d-4d13-8d3e-ca0dd323d337-tls-certs\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.337713 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.337684 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:50.417436 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.417336 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba071e45-db2d-4d13-8d3e-ca0dd323d337-tls-certs\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.417436 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.417386 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-model-cache\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.417436 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.417408 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-home\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.417436 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.417439 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-tmp-dir\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.417798 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.417458 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-dshm\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.417798 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.417483 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7kbj\" (UniqueName: \"kubernetes.io/projected/ba071e45-db2d-4d13-8d3e-ca0dd323d337-kube-api-access-z7kbj\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.417924 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.417853 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-tmp-dir\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.417924 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.417853 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-model-cache\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.417999 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.417928 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-home\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.419995 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.419969 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-dshm\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.420208 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.420188 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba071e45-db2d-4d13-8d3e-ca0dd323d337-tls-certs\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.424371 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.424349 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7kbj\" (UniqueName: \"kubernetes.io/projected/ba071e45-db2d-4d13-8d3e-ca0dd323d337-kube-api-access-z7kbj\") pod \"test-model-dbcba5-kserve-75fb549579-b7wnk\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.558859 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.558828 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:36:50.739566 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.739537 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk"] Jul 15 12:36:50.741376 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:36:50.741346 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba071e45_db2d_4d13_8d3e_ca0dd323d337.slice/crio-b0988ea28bd5e83e8f3f23bb55cc637530b6256835f04624731f6ed38aeef18d WatchSource:0}: Error finding container b0988ea28bd5e83e8f3f23bb55cc637530b6256835f04624731f6ed38aeef18d: Status 404 returned error can't find the container with id b0988ea28bd5e83e8f3f23bb55cc637530b6256835f04624731f6ed38aeef18d Jul 15 12:36:50.982439 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.982392 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" event={"ID":"ba071e45-db2d-4d13-8d3e-ca0dd323d337","Type":"ContainerStarted","Data":"99a5ed4611941c437ac555eac3533076a2d826244d1c5fd108d556dfe7af0b81"} Jul 15 12:36:50.982655 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.982453 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" event={"ID":"ba071e45-db2d-4d13-8d3e-ca0dd323d337","Type":"ContainerStarted","Data":"b0988ea28bd5e83e8f3f23bb55cc637530b6256835f04624731f6ed38aeef18d"} Jul 15 12:36:50.999419 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:50.999310 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" podStartSLOduration=0.999289542 podStartE2EDuration="999.289542ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:36:50.998133316 +0000 UTC m=+2278.411661667" watchObservedRunningTime="2026-07-15 12:36:50.999289542 +0000 UTC m=+2278.412817893" Jul 15 12:36:51.150887 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:51.150839 2570 patch_prober.go:28] interesting pod/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 15 12:36:51.151268 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:51.150906 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" podUID="f22668ef-408e-43a5-914f-953a544649eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 15 12:36:52.150153 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:52.150109 2570 patch_prober.go:28] interesting pod/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 15 12:36:52.150334 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:52.150179 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" podUID="f22668ef-408e-43a5-914f-953a544649eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 15 12:36:53.152890 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:53.152861 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:53.153312 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:53.153293 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:53.153926 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:53.153905 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:36:53.395557 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:53.395527 2570 scope.go:117] "RemoveContainer" containerID="df7aa4759d02a0763fc3d98f91c88d31ab3de296b9670574a1174a84f9b370ab" Jul 15 12:36:54.296119 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:54.296082 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:54.394195 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:36:54.394143 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:00.559217 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:00.559177 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:37:00.559726 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:00.559260 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:37:00.572039 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:00.572015 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:37:01.045897 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:01.045870 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:37:10.286421 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:10.286384 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:24.243445 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:24.243410 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:30.854619 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:30.854586 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk"] Jul 15 12:37:30.871450 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:30.871414 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:31.792781 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:31.792732 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:32.575965 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:32.575929 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:32.676711 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:32.676668 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:42.332335 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:42.332299 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k"] Jul 15 12:37:42.332792 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:42.332536 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" podUID="b1012c92-8790-4ce8-80be-94f12c2ecf81" containerName="maas-api" containerID="cri-o://d39a0773e4c99e92cadcec79f3a42e681c07684a831d8991060dca28603ad4b7" gracePeriod=30 Jul 15 12:37:42.372564 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:42.372534 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:42.585053 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:42.585004 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:37:42.627726 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:42.627694 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1012c92-8790-4ce8-80be-94f12c2ecf81-openshift-service-ca\") pod \"b1012c92-8790-4ce8-80be-94f12c2ecf81\" (UID: \"b1012c92-8790-4ce8-80be-94f12c2ecf81\") " Jul 15 12:37:42.627911 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:42.627803 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l4vx\" (UniqueName: \"kubernetes.io/projected/b1012c92-8790-4ce8-80be-94f12c2ecf81-kube-api-access-9l4vx\") pod \"b1012c92-8790-4ce8-80be-94f12c2ecf81\" (UID: \"b1012c92-8790-4ce8-80be-94f12c2ecf81\") " Jul 15 12:37:42.627911 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:42.627843 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1012c92-8790-4ce8-80be-94f12c2ecf81-maas-api-tls\") pod \"b1012c92-8790-4ce8-80be-94f12c2ecf81\" (UID: \"b1012c92-8790-4ce8-80be-94f12c2ecf81\") " Jul 15 12:37:42.628056 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:42.628032 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1012c92-8790-4ce8-80be-94f12c2ecf81-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b1012c92-8790-4ce8-80be-94f12c2ecf81" (UID: "b1012c92-8790-4ce8-80be-94f12c2ecf81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:37:42.628127 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:42.628115 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1012c92-8790-4ce8-80be-94f12c2ecf81-openshift-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:37:42.630051 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:42.630022 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1012c92-8790-4ce8-80be-94f12c2ecf81-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b1012c92-8790-4ce8-80be-94f12c2ecf81" (UID: "b1012c92-8790-4ce8-80be-94f12c2ecf81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:37:42.630051 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:42.630031 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1012c92-8790-4ce8-80be-94f12c2ecf81-kube-api-access-9l4vx" (OuterVolumeSpecName: "kube-api-access-9l4vx") pod "b1012c92-8790-4ce8-80be-94f12c2ecf81" (UID: "b1012c92-8790-4ce8-80be-94f12c2ecf81"). InnerVolumeSpecName "kube-api-access-9l4vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:37:42.729174 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:42.729145 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1012c92-8790-4ce8-80be-94f12c2ecf81-maas-api-tls\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:37:42.729174 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:42.729175 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l4vx\" (UniqueName: \"kubernetes.io/projected/b1012c92-8790-4ce8-80be-94f12c2ecf81-kube-api-access-9l4vx\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:37:43.250173 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:43.250136 2570 generic.go:356] "Generic (PLEG): container finished" podID="b1012c92-8790-4ce8-80be-94f12c2ecf81" containerID="d39a0773e4c99e92cadcec79f3a42e681c07684a831d8991060dca28603ad4b7" exitCode=0 Jul 15 12:37:43.250173 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:43.250176 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" Jul 15 12:37:43.250376 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:43.250213 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" event={"ID":"b1012c92-8790-4ce8-80be-94f12c2ecf81","Type":"ContainerDied","Data":"d39a0773e4c99e92cadcec79f3a42e681c07684a831d8991060dca28603ad4b7"} Jul 15 12:37:43.250376 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:43.250269 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k" event={"ID":"b1012c92-8790-4ce8-80be-94f12c2ecf81","Type":"ContainerDied","Data":"b4b2d1621b179a217c8cba7ad37fe07a9366c6dad315b0b1f1ca8d1e55b5fbd7"} Jul 15 12:37:43.250376 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:43.250291 2570 scope.go:117] "RemoveContainer" containerID="d39a0773e4c99e92cadcec79f3a42e681c07684a831d8991060dca28603ad4b7" Jul 15 12:37:43.258997 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:43.258973 2570 scope.go:117] "RemoveContainer" containerID="d39a0773e4c99e92cadcec79f3a42e681c07684a831d8991060dca28603ad4b7" Jul 15 12:37:43.259283 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:37:43.259263 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39a0773e4c99e92cadcec79f3a42e681c07684a831d8991060dca28603ad4b7\": container with ID starting with d39a0773e4c99e92cadcec79f3a42e681c07684a831d8991060dca28603ad4b7 not found: ID does not exist" containerID="d39a0773e4c99e92cadcec79f3a42e681c07684a831d8991060dca28603ad4b7" Jul 15 12:37:43.259356 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:43.259292 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39a0773e4c99e92cadcec79f3a42e681c07684a831d8991060dca28603ad4b7"} err="failed to get container status \"d39a0773e4c99e92cadcec79f3a42e681c07684a831d8991060dca28603ad4b7\": rpc error: code = NotFound desc = could not find container \"d39a0773e4c99e92cadcec79f3a42e681c07684a831d8991060dca28603ad4b7\": container with ID starting with d39a0773e4c99e92cadcec79f3a42e681c07684a831d8991060dca28603ad4b7 not found: ID does not exist" Jul 15 12:37:43.265519 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:43.265497 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k"] Jul 15 12:37:43.270949 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:43.270928 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-dbcba5-8765b4888-4b82k"] Jul 15 12:37:45.092221 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:45.092143 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1012c92-8790-4ce8-80be-94f12c2ecf81" path="/var/lib/kubelet/pods/b1012c92-8790-4ce8-80be-94f12c2ecf81/volumes" Jul 15 12:37:45.875509 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:45.875477 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" podUID="ba071e45-db2d-4d13-8d3e-ca0dd323d337" containerName="main" containerID="cri-o://99a5ed4611941c437ac555eac3533076a2d826244d1c5fd108d556dfe7af0b81" gracePeriod=45 Jul 15 12:37:46.130816 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.130755 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:37:46.158272 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.158232 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-dshm\") pod \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " Jul 15 12:37:46.158445 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.158286 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba071e45-db2d-4d13-8d3e-ca0dd323d337-tls-certs\") pod \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " Jul 15 12:37:46.158445 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.158312 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-home\") pod \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " Jul 15 12:37:46.158445 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.158331 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-tmp-dir\") pod \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " Jul 15 12:37:46.158445 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.158380 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-model-cache\") pod \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " Jul 15 12:37:46.158670 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.158516 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7kbj\" (UniqueName: \"kubernetes.io/projected/ba071e45-db2d-4d13-8d3e-ca0dd323d337-kube-api-access-z7kbj\") pod \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\" (UID: \"ba071e45-db2d-4d13-8d3e-ca0dd323d337\") " Jul 15 12:37:46.159187 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.158844 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba071e45-db2d-4d13-8d3e-ca0dd323d337" (UID: "ba071e45-db2d-4d13-8d3e-ca0dd323d337"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:37:46.159187 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.159033 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba071e45-db2d-4d13-8d3e-ca0dd323d337" (UID: "ba071e45-db2d-4d13-8d3e-ca0dd323d337"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:37:46.159187 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.159119 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-home" (OuterVolumeSpecName: "home") pod "ba071e45-db2d-4d13-8d3e-ca0dd323d337" (UID: "ba071e45-db2d-4d13-8d3e-ca0dd323d337"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:37:46.161526 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.161461 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba071e45-db2d-4d13-8d3e-ca0dd323d337-kube-api-access-z7kbj" (OuterVolumeSpecName: "kube-api-access-z7kbj") pod "ba071e45-db2d-4d13-8d3e-ca0dd323d337" (UID: "ba071e45-db2d-4d13-8d3e-ca0dd323d337"). InnerVolumeSpecName "kube-api-access-z7kbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:37:46.161639 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.161527 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba071e45-db2d-4d13-8d3e-ca0dd323d337-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba071e45-db2d-4d13-8d3e-ca0dd323d337" (UID: "ba071e45-db2d-4d13-8d3e-ca0dd323d337"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:37:46.161639 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.161576 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-dshm" (OuterVolumeSpecName: "dshm") pod "ba071e45-db2d-4d13-8d3e-ca0dd323d337" (UID: "ba071e45-db2d-4d13-8d3e-ca0dd323d337"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:37:46.260316 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.260277 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-model-cache\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:37:46.260316 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.260310 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7kbj\" (UniqueName: \"kubernetes.io/projected/ba071e45-db2d-4d13-8d3e-ca0dd323d337-kube-api-access-z7kbj\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:37:46.260316 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.260321 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-dshm\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:37:46.260560 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.260329 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba071e45-db2d-4d13-8d3e-ca0dd323d337-tls-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:37:46.260560 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.260338 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-home\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:37:46.260560 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.260345 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba071e45-db2d-4d13-8d3e-ca0dd323d337-tmp-dir\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:37:46.268366 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.268320 2570 generic.go:356] "Generic (PLEG): container finished" podID="ba071e45-db2d-4d13-8d3e-ca0dd323d337" containerID="99a5ed4611941c437ac555eac3533076a2d826244d1c5fd108d556dfe7af0b81" exitCode=0 Jul 15 12:37:46.268515 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.268377 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" Jul 15 12:37:46.268515 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.268399 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" event={"ID":"ba071e45-db2d-4d13-8d3e-ca0dd323d337","Type":"ContainerDied","Data":"99a5ed4611941c437ac555eac3533076a2d826244d1c5fd108d556dfe7af0b81"} Jul 15 12:37:46.268515 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.268440 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk" event={"ID":"ba071e45-db2d-4d13-8d3e-ca0dd323d337","Type":"ContainerDied","Data":"b0988ea28bd5e83e8f3f23bb55cc637530b6256835f04624731f6ed38aeef18d"} Jul 15 12:37:46.268515 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.268456 2570 scope.go:117] "RemoveContainer" containerID="99a5ed4611941c437ac555eac3533076a2d826244d1c5fd108d556dfe7af0b81" Jul 15 12:37:46.277293 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.277271 2570 scope.go:117] "RemoveContainer" containerID="99a5ed4611941c437ac555eac3533076a2d826244d1c5fd108d556dfe7af0b81" Jul 15 12:37:46.277568 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:37:46.277545 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a5ed4611941c437ac555eac3533076a2d826244d1c5fd108d556dfe7af0b81\": container with ID starting with 99a5ed4611941c437ac555eac3533076a2d826244d1c5fd108d556dfe7af0b81 not found: ID does not exist" containerID="99a5ed4611941c437ac555eac3533076a2d826244d1c5fd108d556dfe7af0b81" Jul 15 12:37:46.277630 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.277578 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a5ed4611941c437ac555eac3533076a2d826244d1c5fd108d556dfe7af0b81"} err="failed to get container status \"99a5ed4611941c437ac555eac3533076a2d826244d1c5fd108d556dfe7af0b81\": rpc error: code = NotFound desc = could not find container \"99a5ed4611941c437ac555eac3533076a2d826244d1c5fd108d556dfe7af0b81\": container with ID starting with 99a5ed4611941c437ac555eac3533076a2d826244d1c5fd108d556dfe7af0b81 not found: ID does not exist" Jul 15 12:37:46.288616 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.288592 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk"] Jul 15 12:37:46.294703 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:46.294682 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-dbcba5/test-model-dbcba5-kserve-75fb549579-b7wnk"] Jul 15 12:37:47.015763 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:47.015703 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:47.091922 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:47.091887 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba071e45-db2d-4d13-8d3e-ca0dd323d337" path="/var/lib/kubelet/pods/ba071e45-db2d-4d13-8d3e-ca0dd323d337/volumes" Jul 15 12:37:52.928644 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:52.928608 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:53.028811 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:53.028764 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:57.684634 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:57.684595 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h"] Jul 15 12:37:57.685151 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:57.684893 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" podUID="13289992-9721-473f-8b4a-7237d7db78fe" containerName="istio-proxy" containerID="cri-o://f8c964563c0cbb63c0c804aa1dd11fe9f493c8573354bffff9ae5698d96284b8" gracePeriod=30 Jul 15 12:37:57.715179 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:57.715146 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:57.952853 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:57.952682 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:58.064370 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.064327 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt"] Jul 15 12:37:58.064912 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.064894 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1012c92-8790-4ce8-80be-94f12c2ecf81" containerName="maas-api" Jul 15 12:37:58.065002 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.064915 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1012c92-8790-4ce8-80be-94f12c2ecf81" containerName="maas-api" Jul 15 12:37:58.065002 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.064942 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba071e45-db2d-4d13-8d3e-ca0dd323d337" containerName="main" Jul 15 12:37:58.065002 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.064950 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba071e45-db2d-4d13-8d3e-ca0dd323d337" containerName="main" Jul 15 12:37:58.065143 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.065067 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1012c92-8790-4ce8-80be-94f12c2ecf81" containerName="maas-api" Jul 15 12:37:58.065143 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.065077 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba071e45-db2d-4d13-8d3e-ca0dd323d337" containerName="main" Jul 15 12:37:58.068443 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.068422 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt" Jul 15 12:37:58.077274 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.077242 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt"] Jul 15 12:37:58.082984 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.082958 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tf4tp\"" Jul 15 12:37:58.169759 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.169692 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b74f1452-8689-4e60-8075-b6ab99ce5d62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt\" (UID: \"b74f1452-8689-4e60-8075-b6ab99ce5d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt" Jul 15 12:37:58.271110 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.271018 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b74f1452-8689-4e60-8075-b6ab99ce5d62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt\" (UID: \"b74f1452-8689-4e60-8075-b6ab99ce5d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt" Jul 15 12:37:58.271684 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.271658 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b74f1452-8689-4e60-8075-b6ab99ce5d62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt\" (UID: \"b74f1452-8689-4e60-8075-b6ab99ce5d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt" Jul 15 12:37:58.386806 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.386775 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt" Jul 15 12:37:58.522430 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.522377 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt"] Jul 15 12:37:58.525030 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:37:58.525003 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74f1452_8689_4e60_8075_b6ab99ce5d62.slice/crio-066336e362308f082da9fe74a8f2063ab93c3313a7ad7fb8a340b0107294e832 WatchSource:0}: Error finding container 066336e362308f082da9fe74a8f2063ab93c3313a7ad7fb8a340b0107294e832: Status 404 returned error can't find the container with id 066336e362308f082da9fe74a8f2063ab93c3313a7ad7fb8a340b0107294e832 Jul 15 12:37:58.665619 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.665586 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:58.766924 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:58.766896 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:59.342267 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:59.342232 2570 generic.go:356] "Generic (PLEG): container finished" podID="b74f1452-8689-4e60-8075-b6ab99ce5d62" containerID="4351b236fe42da6dceefb2b07c1a2e943b482cdf05b52d115c5508f5be73e1c4" exitCode=0 Jul 15 12:37:59.342438 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:59.342318 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt" event={"ID":"b74f1452-8689-4e60-8075-b6ab99ce5d62","Type":"ContainerDied","Data":"4351b236fe42da6dceefb2b07c1a2e943b482cdf05b52d115c5508f5be73e1c4"} Jul 15 12:37:59.342438 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:37:59.342366 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt" event={"ID":"b74f1452-8689-4e60-8075-b6ab99ce5d62","Type":"ContainerStarted","Data":"066336e362308f082da9fe74a8f2063ab93c3313a7ad7fb8a340b0107294e832"} Jul 15 12:38:00.479998 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:00.479965 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt" Jul 15 12:38:00.595610 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:00.595574 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b74f1452-8689-4e60-8075-b6ab99ce5d62-maas-api-service-ca\") pod \"b74f1452-8689-4e60-8075-b6ab99ce5d62\" (UID: \"b74f1452-8689-4e60-8075-b6ab99ce5d62\") " Jul 15 12:38:00.595944 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:00.595923 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74f1452-8689-4e60-8075-b6ab99ce5d62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b74f1452-8689-4e60-8075-b6ab99ce5d62" (UID: "b74f1452-8689-4e60-8075-b6ab99ce5d62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:38:00.697241 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:00.697153 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b74f1452-8689-4e60-8075-b6ab99ce5d62-maas-api-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:01.354216 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:01.354141 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt" Jul 15 12:38:01.354216 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:01.354160 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt" event={"ID":"b74f1452-8689-4e60-8075-b6ab99ce5d62","Type":"ContainerDied","Data":"066336e362308f082da9fe74a8f2063ab93c3313a7ad7fb8a340b0107294e832"} Jul 15 12:38:01.354216 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:01.354208 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="066336e362308f082da9fe74a8f2063ab93c3313a7ad7fb8a340b0107294e832" Jul 15 12:38:01.532810 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:01.532769 2570 patch_prober.go:28] interesting pod/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 12:38:01.533180 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:01.532825 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" podUID="13289992-9721-473f-8b4a-7237d7db78fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 12:38:02.946333 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:02.946269 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:38:03.017931 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.017898 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13289992-9721-473f-8b4a-7237d7db78fe-istio-token\") pod \"13289992-9721-473f-8b4a-7237d7db78fe\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " Jul 15 12:38:03.018139 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.017940 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-workload-certs\") pod \"13289992-9721-473f-8b4a-7237d7db78fe\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " Jul 15 12:38:03.018139 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.017965 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwjg9\" (UniqueName: \"kubernetes.io/projected/13289992-9721-473f-8b4a-7237d7db78fe-kube-api-access-mwjg9\") pod \"13289992-9721-473f-8b4a-7237d7db78fe\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " Jul 15 12:38:03.018139 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.017988 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-credential-socket\") pod \"13289992-9721-473f-8b4a-7237d7db78fe\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " Jul 15 12:38:03.018139 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.018088 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13289992-9721-473f-8b4a-7237d7db78fe-istiod-ca-cert\") pod \"13289992-9721-473f-8b4a-7237d7db78fe\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " Jul 15 12:38:03.018139 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.018118 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-istio-data\") pod \"13289992-9721-473f-8b4a-7237d7db78fe\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " Jul 15 12:38:03.018402 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.018153 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13289992-9721-473f-8b4a-7237d7db78fe-istio-podinfo\") pod \"13289992-9721-473f-8b4a-7237d7db78fe\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " Jul 15 12:38:03.018402 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.018206 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-istio-envoy\") pod \"13289992-9721-473f-8b4a-7237d7db78fe\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " Jul 15 12:38:03.018402 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.018238 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-workload-socket\") pod \"13289992-9721-473f-8b4a-7237d7db78fe\" (UID: \"13289992-9721-473f-8b4a-7237d7db78fe\") " Jul 15 12:38:03.018402 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.018239 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "13289992-9721-473f-8b4a-7237d7db78fe" (UID: "13289992-9721-473f-8b4a-7237d7db78fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:38:03.018605 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.018451 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "13289992-9721-473f-8b4a-7237d7db78fe" (UID: "13289992-9721-473f-8b4a-7237d7db78fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:38:03.018605 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.018537 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13289992-9721-473f-8b4a-7237d7db78fe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "13289992-9721-473f-8b4a-7237d7db78fe" (UID: "13289992-9721-473f-8b4a-7237d7db78fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:38:03.018605 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.018593 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-workload-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:03.018795 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.018647 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-credential-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:03.018795 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.018693 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "13289992-9721-473f-8b4a-7237d7db78fe" (UID: "13289992-9721-473f-8b4a-7237d7db78fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:38:03.019463 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.019433 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-istio-data" (OuterVolumeSpecName: "istio-data") pod "13289992-9721-473f-8b4a-7237d7db78fe" (UID: "13289992-9721-473f-8b4a-7237d7db78fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:38:03.020466 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.020439 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13289992-9721-473f-8b4a-7237d7db78fe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "13289992-9721-473f-8b4a-7237d7db78fe" (UID: "13289992-9721-473f-8b4a-7237d7db78fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:38:03.020591 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.020561 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13289992-9721-473f-8b4a-7237d7db78fe-kube-api-access-mwjg9" (OuterVolumeSpecName: "kube-api-access-mwjg9") pod "13289992-9721-473f-8b4a-7237d7db78fe" (UID: "13289992-9721-473f-8b4a-7237d7db78fe"). InnerVolumeSpecName "kube-api-access-mwjg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:38:03.020942 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.020912 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "13289992-9721-473f-8b4a-7237d7db78fe" (UID: "13289992-9721-473f-8b4a-7237d7db78fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:38:03.020942 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.020924 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13289992-9721-473f-8b4a-7237d7db78fe-istio-token" (OuterVolumeSpecName: "istio-token") pod "13289992-9721-473f-8b4a-7237d7db78fe" (UID: "13289992-9721-473f-8b4a-7237d7db78fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:38:03.119660 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.119634 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13289992-9721-473f-8b4a-7237d7db78fe-istiod-ca-cert\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:03.119660 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.119660 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-istio-data\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:03.119660 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.119668 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13289992-9721-473f-8b4a-7237d7db78fe-istio-podinfo\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:03.119895 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.119676 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-istio-envoy\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:03.119895 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.119684 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13289992-9721-473f-8b4a-7237d7db78fe-workload-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:03.119895 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.119691 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13289992-9721-473f-8b4a-7237d7db78fe-istio-token\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:03.119895 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.119699 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwjg9\" (UniqueName: \"kubernetes.io/projected/13289992-9721-473f-8b4a-7237d7db78fe-kube-api-access-mwjg9\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:03.364721 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.364693 2570 generic.go:356] "Generic (PLEG): container finished" podID="13289992-9721-473f-8b4a-7237d7db78fe" containerID="f8c964563c0cbb63c0c804aa1dd11fe9f493c8573354bffff9ae5698d96284b8" exitCode=0 Jul 15 12:38:03.364920 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.364767 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" Jul 15 12:38:03.364920 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.364783 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" event={"ID":"13289992-9721-473f-8b4a-7237d7db78fe","Type":"ContainerDied","Data":"f8c964563c0cbb63c0c804aa1dd11fe9f493c8573354bffff9ae5698d96284b8"} Jul 15 12:38:03.364920 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.364845 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h" event={"ID":"13289992-9721-473f-8b4a-7237d7db78fe","Type":"ContainerDied","Data":"3a9d19977d909dfcf50039844f8aee498f9ed3d424dae2e15adad628620e636f"} Jul 15 12:38:03.364920 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.364875 2570 scope.go:117] "RemoveContainer" containerID="f8c964563c0cbb63c0c804aa1dd11fe9f493c8573354bffff9ae5698d96284b8" Jul 15 12:38:03.373622 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.373603 2570 scope.go:117] "RemoveContainer" containerID="6e8059c1d5aa9b9772ee7d237e75f4ac9b94b577d131e736ca1a97bc2d18f6f2" Jul 15 12:38:03.381895 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.381867 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h"] Jul 15 12:38:03.382670 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.382650 2570 scope.go:117] "RemoveContainer" containerID="f8c964563c0cbb63c0c804aa1dd11fe9f493c8573354bffff9ae5698d96284b8" Jul 15 12:38:03.382956 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:38:03.382937 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c964563c0cbb63c0c804aa1dd11fe9f493c8573354bffff9ae5698d96284b8\": container with ID starting with f8c964563c0cbb63c0c804aa1dd11fe9f493c8573354bffff9ae5698d96284b8 not found: ID does not exist" containerID="f8c964563c0cbb63c0c804aa1dd11fe9f493c8573354bffff9ae5698d96284b8" Jul 15 12:38:03.383061 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.382967 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c964563c0cbb63c0c804aa1dd11fe9f493c8573354bffff9ae5698d96284b8"} err="failed to get container status \"f8c964563c0cbb63c0c804aa1dd11fe9f493c8573354bffff9ae5698d96284b8\": rpc error: code = NotFound desc = could not find container \"f8c964563c0cbb63c0c804aa1dd11fe9f493c8573354bffff9ae5698d96284b8\": container with ID starting with f8c964563c0cbb63c0c804aa1dd11fe9f493c8573354bffff9ae5698d96284b8 not found: ID does not exist" Jul 15 12:38:03.383061 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.382990 2570 scope.go:117] "RemoveContainer" containerID="6e8059c1d5aa9b9772ee7d237e75f4ac9b94b577d131e736ca1a97bc2d18f6f2" Jul 15 12:38:03.383256 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:38:03.383236 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8059c1d5aa9b9772ee7d237e75f4ac9b94b577d131e736ca1a97bc2d18f6f2\": container with ID starting with 6e8059c1d5aa9b9772ee7d237e75f4ac9b94b577d131e736ca1a97bc2d18f6f2 not found: ID does not exist" containerID="6e8059c1d5aa9b9772ee7d237e75f4ac9b94b577d131e736ca1a97bc2d18f6f2" Jul 15 12:38:03.383302 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.383262 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8059c1d5aa9b9772ee7d237e75f4ac9b94b577d131e736ca1a97bc2d18f6f2"} err="failed to get container status \"6e8059c1d5aa9b9772ee7d237e75f4ac9b94b577d131e736ca1a97bc2d18f6f2\": rpc error: code = NotFound desc = could not find container \"6e8059c1d5aa9b9772ee7d237e75f4ac9b94b577d131e736ca1a97bc2d18f6f2\": container with ID starting with 6e8059c1d5aa9b9772ee7d237e75f4ac9b94b577d131e736ca1a97bc2d18f6f2 not found: ID does not exist" Jul 15 12:38:03.388430 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:03.388409 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-dbcba5-openshift-default-554cb998b7-4vs7h"] Jul 15 12:38:05.091781 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:05.091719 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13289992-9721-473f-8b4a-7237d7db78fe" path="/var/lib/kubelet/pods/13289992-9721-473f-8b4a-7237d7db78fe/volumes" Jul 15 12:38:08.498442 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:08.498406 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:13.172207 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:13.172168 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:19.466602 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:19.466564 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:19.563872 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:19.563839 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:23.868666 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:23.868628 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm"] Jul 15 12:38:23.869128 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:23.868866 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" podUID="f22668ef-408e-43a5-914f-953a544649eb" containerName="istio-proxy" containerID="cri-o://93563daf4f80cc6f2e48c23433df0453a96e20525b685270d17f39a76ea0db59" gracePeriod=30 Jul 15 12:38:23.900502 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:23.900469 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:24.132565 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.132483 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:24.484886 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.484808 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:24.760689 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.760609 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k"] Jul 15 12:38:24.761197 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.761171 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b74f1452-8689-4e60-8075-b6ab99ce5d62" containerName="revoke-keys" Jul 15 12:38:24.761197 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.761196 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b74f1452-8689-4e60-8075-b6ab99ce5d62" containerName="revoke-keys" Jul 15 12:38:24.761346 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.761217 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13289992-9721-473f-8b4a-7237d7db78fe" containerName="istio-proxy" Jul 15 12:38:24.761346 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.761223 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="13289992-9721-473f-8b4a-7237d7db78fe" containerName="istio-proxy" Jul 15 12:38:24.761346 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.761284 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b74f1452-8689-4e60-8075-b6ab99ce5d62" containerName="revoke-keys" Jul 15 12:38:24.761346 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.761295 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="13289992-9721-473f-8b4a-7237d7db78fe" containerName="istio-proxy" Jul 15 12:38:24.765858 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.765839 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.768081 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.768063 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-858a49-openshift-default-dockercfg-n9wjl\"" Jul 15 12:38:24.779938 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.779911 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k"] Jul 15 12:38:24.792949 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.792922 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:24.817862 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.817827 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgrrk\" (UniqueName: \"kubernetes.io/projected/b059b0db-2cd4-4677-a1ec-909698f2feee-kube-api-access-rgrrk\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.818045 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.817919 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-data\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.818045 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.817968 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-podinfo\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.818045 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.818005 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-workload-certs\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.818045 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.818031 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-token\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.818229 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.818052 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b059b0db-2cd4-4677-a1ec-909698f2feee-istiod-ca-cert\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.818229 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.818136 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-credential-socket\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.818229 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.818173 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-workload-socket\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.818229 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.818191 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-envoy\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.919071 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.919035 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-workload-certs\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.919071 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.919074 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-token\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.919580 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.919098 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b059b0db-2cd4-4677-a1ec-909698f2feee-istiod-ca-cert\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.919580 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.919132 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-credential-socket\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.919580 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.919154 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-workload-socket\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.919580 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.919169 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-envoy\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.919580 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.919215 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgrrk\" (UniqueName: \"kubernetes.io/projected/b059b0db-2cd4-4677-a1ec-909698f2feee-kube-api-access-rgrrk\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.919580 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.919269 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-data\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.919580 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.919315 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-podinfo\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.919580 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.919520 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-workload-certs\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.920056 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.919589 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-workload-socket\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.920056 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.919642 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-credential-socket\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.920056 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.919769 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-data\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.920177 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.920061 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b059b0db-2cd4-4677-a1ec-909698f2feee-istiod-ca-cert\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.921629 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.921603 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-envoy\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.921861 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.921843 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-podinfo\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.926388 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.926364 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-token\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:24.926560 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:24.926542 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgrrk\" (UniqueName: \"kubernetes.io/projected/b059b0db-2cd4-4677-a1ec-909698f2feee-kube-api-access-rgrrk\") pod \"e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:25.075267 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:25.075182 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:25.076167 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:25.076149 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:25.247633 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:25.247609 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k"] Jul 15 12:38:25.250209 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:38:25.250180 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb059b0db_2cd4_4677_a1ec_909698f2feee.slice/crio-88785008b82d54343756b00c48d9f14c4beaece193d2dd3b6e02cff1b69c77d9 WatchSource:0}: Error finding container 88785008b82d54343756b00c48d9f14c4beaece193d2dd3b6e02cff1b69c77d9: Status 404 returned error can't find the container with id 88785008b82d54343756b00c48d9f14c4beaece193d2dd3b6e02cff1b69c77d9 Jul 15 12:38:25.252409 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:25.252377 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:38:25.252484 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:25.252445 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:38:25.252484 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:25.252475 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:38:25.484428 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:25.484385 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" event={"ID":"b059b0db-2cd4-4677-a1ec-909698f2feee","Type":"ContainerStarted","Data":"161503147e00d9ad004638741f6c7a137c546c2450991765d8ec609cd3230034"} Jul 15 12:38:25.484636 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:25.484459 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" event={"ID":"b059b0db-2cd4-4677-a1ec-909698f2feee","Type":"ContainerStarted","Data":"88785008b82d54343756b00c48d9f14c4beaece193d2dd3b6e02cff1b69c77d9"} Jul 15 12:38:25.506237 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:25.506131 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" podStartSLOduration=1.506109414 podStartE2EDuration="1.506109414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:38:25.50550045 +0000 UTC m=+2372.919028801" watchObservedRunningTime="2026-07-15 12:38:25.506109414 +0000 UTC m=+2372.919637766" Jul 15 12:38:26.076438 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:26.076406 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:26.081469 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:26.081444 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:26.495167 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:26.495135 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:26.496248 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:26.496225 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:27.427518 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.427482 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:27.533229 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.533192 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:27.661291 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.661258 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p"] Jul 15 12:38:27.661638 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.661627 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13289992-9721-473f-8b4a-7237d7db78fe" containerName="istio-proxy" Jul 15 12:38:27.661680 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.661640 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="13289992-9721-473f-8b4a-7237d7db78fe" containerName="istio-proxy" Jul 15 12:38:27.661719 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.661714 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="13289992-9721-473f-8b4a-7237d7db78fe" containerName="istio-proxy" Jul 15 12:38:27.665432 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.665413 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:27.667280 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.667259 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-858a49-serving-cert\"" Jul 15 12:38:27.676893 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.676870 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p"] Jul 15 12:38:27.694232 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.694145 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:27.747151 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.747109 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfz8p\" (UniqueName: \"kubernetes.io/projected/57b16b4c-781c-4247-87da-0c0f9064f112-kube-api-access-qfz8p\") pod \"maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p\" (UID: \"57b16b4c-781c-4247-87da-0c0f9064f112\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:27.747328 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.747163 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b16b4c-781c-4247-87da-0c0f9064f112-maas-api-tls\") pod \"maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p\" (UID: \"57b16b4c-781c-4247-87da-0c0f9064f112\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:27.747328 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.747226 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b16b4c-781c-4247-87da-0c0f9064f112-openshift-service-ca\") pod \"maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p\" (UID: \"57b16b4c-781c-4247-87da-0c0f9064f112\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:27.848181 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.848143 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfz8p\" (UniqueName: \"kubernetes.io/projected/57b16b4c-781c-4247-87da-0c0f9064f112-kube-api-access-qfz8p\") pod \"maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p\" (UID: \"57b16b4c-781c-4247-87da-0c0f9064f112\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:27.848362 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.848199 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b16b4c-781c-4247-87da-0c0f9064f112-maas-api-tls\") pod \"maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p\" (UID: \"57b16b4c-781c-4247-87da-0c0f9064f112\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:27.848362 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.848231 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b16b4c-781c-4247-87da-0c0f9064f112-openshift-service-ca\") pod \"maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p\" (UID: \"57b16b4c-781c-4247-87da-0c0f9064f112\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:27.848451 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:38:27.848369 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-serving-cert: secret "maas-api-e2e-api-a-858a49-serving-cert" not found Jul 15 12:38:27.848498 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:38:27.848450 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57b16b4c-781c-4247-87da-0c0f9064f112-maas-api-tls podName:57b16b4c-781c-4247-87da-0c0f9064f112 nodeName:}" failed. No retries permitted until 2026-07-15 12:38:28.34842912 +0000 UTC m=+2375.761957451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57b16b4c-781c-4247-87da-0c0f9064f112-maas-api-tls") pod "maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" (UID: "57b16b4c-781c-4247-87da-0c0f9064f112") : secret "maas-api-e2e-api-a-858a49-serving-cert" not found Jul 15 12:38:27.849002 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.848981 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b16b4c-781c-4247-87da-0c0f9064f112-openshift-service-ca\") pod \"maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p\" (UID: \"57b16b4c-781c-4247-87da-0c0f9064f112\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:27.858785 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:27.858726 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfz8p\" (UniqueName: \"kubernetes.io/projected/57b16b4c-781c-4247-87da-0c0f9064f112-kube-api-access-qfz8p\") pod \"maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p\" (UID: \"57b16b4c-781c-4247-87da-0c0f9064f112\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:28.270055 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:28.270023 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:28.353607 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:28.353572 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b16b4c-781c-4247-87da-0c0f9064f112-maas-api-tls\") pod \"maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p\" (UID: \"57b16b4c-781c-4247-87da-0c0f9064f112\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:28.356198 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:28.356174 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b16b4c-781c-4247-87da-0c0f9064f112-maas-api-tls\") pod \"maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p\" (UID: \"57b16b4c-781c-4247-87da-0c0f9064f112\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:28.376702 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:28.376672 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:28.578295 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:28.578215 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:28.718673 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:28.718633 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p"] Jul 15 12:38:28.719865 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:38:28.719841 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b16b4c_781c_4247_87da_0c0f9064f112.slice/crio-57d52f76f77fa51330d460095b3e936608a7fd5d936e69c19262c6d7a462fae7 WatchSource:0}: Error finding container 57d52f76f77fa51330d460095b3e936608a7fd5d936e69c19262c6d7a462fae7: Status 404 returned error can't find the container with id 57d52f76f77fa51330d460095b3e936608a7fd5d936e69c19262c6d7a462fae7 Jul 15 12:38:28.721440 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:28.721420 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:38:29.126840 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.126815 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:38:29.161587 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.161510 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22668ef-408e-43a5-914f-953a544649eb-istio-token\") pod \"f22668ef-408e-43a5-914f-953a544649eb\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " Jul 15 12:38:29.161587 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.161579 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgdm5\" (UniqueName: \"kubernetes.io/projected/f22668ef-408e-43a5-914f-953a544649eb-kube-api-access-jgdm5\") pod \"f22668ef-408e-43a5-914f-953a544649eb\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " Jul 15 12:38:29.161832 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.161600 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22668ef-408e-43a5-914f-953a544649eb-istiod-ca-cert\") pod \"f22668ef-408e-43a5-914f-953a544649eb\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " Jul 15 12:38:29.161832 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.161627 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-credential-socket\") pod \"f22668ef-408e-43a5-914f-953a544649eb\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " Jul 15 12:38:29.161832 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.161666 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-workload-socket\") pod \"f22668ef-408e-43a5-914f-953a544649eb\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " Jul 15 12:38:29.161832 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.161734 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-workload-certs\") pod \"f22668ef-408e-43a5-914f-953a544649eb\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " Jul 15 12:38:29.161832 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.161786 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-istio-data\") pod \"f22668ef-408e-43a5-914f-953a544649eb\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " Jul 15 12:38:29.161832 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.161826 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-istio-envoy\") pod \"f22668ef-408e-43a5-914f-953a544649eb\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " Jul 15 12:38:29.162203 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.161857 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22668ef-408e-43a5-914f-953a544649eb-istio-podinfo\") pod \"f22668ef-408e-43a5-914f-953a544649eb\" (UID: \"f22668ef-408e-43a5-914f-953a544649eb\") " Jul 15 12:38:29.162203 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.161968 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f22668ef-408e-43a5-914f-953a544649eb" (UID: "f22668ef-408e-43a5-914f-953a544649eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:38:29.162203 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.162009 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f22668ef-408e-43a5-914f-953a544649eb" (UID: "f22668ef-408e-43a5-914f-953a544649eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:38:29.162203 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.162014 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22668ef-408e-43a5-914f-953a544649eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f22668ef-408e-43a5-914f-953a544649eb" (UID: "f22668ef-408e-43a5-914f-953a544649eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:38:29.162203 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.162039 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f22668ef-408e-43a5-914f-953a544649eb" (UID: "f22668ef-408e-43a5-914f-953a544649eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:38:29.162625 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.162264 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-workload-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:29.162625 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.162285 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22668ef-408e-43a5-914f-953a544649eb-istiod-ca-cert\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:29.162625 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.162346 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-credential-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:29.162625 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.162361 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-workload-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:29.162873 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.162851 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "f22668ef-408e-43a5-914f-953a544649eb" (UID: "f22668ef-408e-43a5-914f-953a544649eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:38:29.164148 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.164120 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22668ef-408e-43a5-914f-953a544649eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "f22668ef-408e-43a5-914f-953a544649eb" (UID: "f22668ef-408e-43a5-914f-953a544649eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:38:29.164594 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.164578 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f22668ef-408e-43a5-914f-953a544649eb" (UID: "f22668ef-408e-43a5-914f-953a544649eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:38:29.164698 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.164675 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f22668ef-408e-43a5-914f-953a544649eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f22668ef-408e-43a5-914f-953a544649eb" (UID: "f22668ef-408e-43a5-914f-953a544649eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:38:29.164903 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.164881 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22668ef-408e-43a5-914f-953a544649eb-kube-api-access-jgdm5" (OuterVolumeSpecName: "kube-api-access-jgdm5") pod "f22668ef-408e-43a5-914f-953a544649eb" (UID: "f22668ef-408e-43a5-914f-953a544649eb"). InnerVolumeSpecName "kube-api-access-jgdm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:38:29.263292 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.263252 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-istio-envoy\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:29.263292 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.263292 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22668ef-408e-43a5-914f-953a544649eb-istio-podinfo\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:29.263470 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.263305 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22668ef-408e-43a5-914f-953a544649eb-istio-token\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:29.263470 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.263320 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgdm5\" (UniqueName: \"kubernetes.io/projected/f22668ef-408e-43a5-914f-953a544649eb-kube-api-access-jgdm5\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:29.263470 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.263334 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22668ef-408e-43a5-914f-953a544649eb-istio-data\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:29.513462 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.513389 2570 generic.go:356] "Generic (PLEG): container finished" podID="f22668ef-408e-43a5-914f-953a544649eb" containerID="93563daf4f80cc6f2e48c23433df0453a96e20525b685270d17f39a76ea0db59" exitCode=0 Jul 15 12:38:29.513462 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.513437 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" Jul 15 12:38:29.513654 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.513472 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" event={"ID":"f22668ef-408e-43a5-914f-953a544649eb","Type":"ContainerDied","Data":"93563daf4f80cc6f2e48c23433df0453a96e20525b685270d17f39a76ea0db59"} Jul 15 12:38:29.513654 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.513526 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm" event={"ID":"f22668ef-408e-43a5-914f-953a544649eb","Type":"ContainerDied","Data":"7d905ba362fcb4102d8266a85b3ba95506d6827ee483bdf03e3ba743c65db53d"} Jul 15 12:38:29.513654 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.513558 2570 scope.go:117] "RemoveContainer" containerID="93563daf4f80cc6f2e48c23433df0453a96e20525b685270d17f39a76ea0db59" Jul 15 12:38:29.515092 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.515042 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" event={"ID":"57b16b4c-781c-4247-87da-0c0f9064f112","Type":"ContainerStarted","Data":"211a0610adbb312d616343629321a37873d469d9da4d05ee57b380450abadfdc"} Jul 15 12:38:29.515231 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.515098 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" event={"ID":"57b16b4c-781c-4247-87da-0c0f9064f112","Type":"ContainerStarted","Data":"57d52f76f77fa51330d460095b3e936608a7fd5d936e69c19262c6d7a462fae7"} Jul 15 12:38:29.515231 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.515184 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:29.523234 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.523218 2570 scope.go:117] "RemoveContainer" containerID="3029a8c559aab996f3b2848bcbd3aa614e73f34b8bd8a6436cda7d3dbe19d6e7" Jul 15 12:38:29.534402 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.533399 2570 scope.go:117] "RemoveContainer" containerID="93563daf4f80cc6f2e48c23433df0453a96e20525b685270d17f39a76ea0db59" Jul 15 12:38:29.534599 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:38:29.534571 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93563daf4f80cc6f2e48c23433df0453a96e20525b685270d17f39a76ea0db59\": container with ID starting with 93563daf4f80cc6f2e48c23433df0453a96e20525b685270d17f39a76ea0db59 not found: ID does not exist" containerID="93563daf4f80cc6f2e48c23433df0453a96e20525b685270d17f39a76ea0db59" Jul 15 12:38:29.534705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.534610 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93563daf4f80cc6f2e48c23433df0453a96e20525b685270d17f39a76ea0db59"} err="failed to get container status \"93563daf4f80cc6f2e48c23433df0453a96e20525b685270d17f39a76ea0db59\": rpc error: code = NotFound desc = could not find container \"93563daf4f80cc6f2e48c23433df0453a96e20525b685270d17f39a76ea0db59\": container with ID starting with 93563daf4f80cc6f2e48c23433df0453a96e20525b685270d17f39a76ea0db59 not found: ID does not exist" Jul 15 12:38:29.534705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.534635 2570 scope.go:117] "RemoveContainer" containerID="3029a8c559aab996f3b2848bcbd3aa614e73f34b8bd8a6436cda7d3dbe19d6e7" Jul 15 12:38:29.534999 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:38:29.534976 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3029a8c559aab996f3b2848bcbd3aa614e73f34b8bd8a6436cda7d3dbe19d6e7\": container with ID starting with 3029a8c559aab996f3b2848bcbd3aa614e73f34b8bd8a6436cda7d3dbe19d6e7 not found: ID does not exist" containerID="3029a8c559aab996f3b2848bcbd3aa614e73f34b8bd8a6436cda7d3dbe19d6e7" Jul 15 12:38:29.535051 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.535006 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3029a8c559aab996f3b2848bcbd3aa614e73f34b8bd8a6436cda7d3dbe19d6e7"} err="failed to get container status \"3029a8c559aab996f3b2848bcbd3aa614e73f34b8bd8a6436cda7d3dbe19d6e7\": rpc error: code = NotFound desc = could not find container \"3029a8c559aab996f3b2848bcbd3aa614e73f34b8bd8a6436cda7d3dbe19d6e7\": container with ID starting with 3029a8c559aab996f3b2848bcbd3aa614e73f34b8bd8a6436cda7d3dbe19d6e7 not found: ID does not exist" Jul 15 12:38:29.536654 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.536614 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" podStartSLOduration=2.074681697 podStartE2EDuration="2.536602301s" totalImagesPullingTime="461.920604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:38:29.533306657 +0000 UTC m=+2376.946835029" watchObservedRunningTime="2026-07-15 12:38:29.536602301 +0000 UTC m=+2376.950130652" Jul 15 12:38:29.545805 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.545783 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm"] Jul 15 12:38:29.549307 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:29.549283 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6dba6e-openshift-default-5d77bfd575-zmzkm"] Jul 15 12:38:31.091573 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:31.091540 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f22668ef-408e-43a5-914f-953a544649eb" path="/var/lib/kubelet/pods/f22668ef-408e-43a5-914f-953a544649eb/volumes" Jul 15 12:38:31.977879 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:31.977839 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:32.465731 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.465692 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:32.751578 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.751498 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s"] Jul 15 12:38:32.751911 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.751898 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f22668ef-408e-43a5-914f-953a544649eb" containerName="istio-proxy" Jul 15 12:38:32.751966 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.751913 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f22668ef-408e-43a5-914f-953a544649eb" containerName="istio-proxy" Jul 15 12:38:32.751966 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.751929 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f22668ef-408e-43a5-914f-953a544649eb" containerName="istio-proxy" Jul 15 12:38:32.751966 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.751935 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f22668ef-408e-43a5-914f-953a544649eb" containerName="istio-proxy" Jul 15 12:38:32.752063 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.751989 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f22668ef-408e-43a5-914f-953a544649eb" containerName="istio-proxy" Jul 15 12:38:32.752063 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.751999 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f22668ef-408e-43a5-914f-953a544649eb" containerName="istio-proxy" Jul 15 12:38:32.756610 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.756588 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.758555 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.758535 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-137ff6-openshift-default-dockercfg-wrglr\"" Jul 15 12:38:32.766800 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.766774 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s"] Jul 15 12:38:32.798407 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.798375 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-token\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.798596 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.798446 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-data\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.798596 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.798478 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-envoy\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.798596 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.798537 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-credential-socket\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.798596 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.798556 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-podinfo\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.798796 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.798612 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-workload-certs\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.798796 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.798698 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnwzt\" (UniqueName: \"kubernetes.io/projected/f52f9329-318f-454d-8e7b-b99d13b1ae8b-kube-api-access-gnwzt\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.798796 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.798781 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-workload-socket\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.798892 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.798811 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istiod-ca-cert\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.900266 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.900228 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istiod-ca-cert\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.900266 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.900271 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-token\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.900540 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.900306 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-data\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.900540 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.900328 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-envoy\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.900540 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.900370 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-credential-socket\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.900540 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.900387 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-podinfo\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.900540 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.900415 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-workload-certs\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.900540 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.900487 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwzt\" (UniqueName: \"kubernetes.io/projected/f52f9329-318f-454d-8e7b-b99d13b1ae8b-kube-api-access-gnwzt\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.900540 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.900541 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-workload-socket\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.900938 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.900902 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-workload-certs\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.900999 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.900983 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-credential-socket\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.901052 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.900991 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-data\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.901052 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.901021 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-workload-socket\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.901422 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.901371 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istiod-ca-cert\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.903065 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.903032 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-envoy\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.903209 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.903080 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-podinfo\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.907580 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.907554 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-token\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:32.907709 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:32.907693 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnwzt\" (UniqueName: \"kubernetes.io/projected/f52f9329-318f-454d-8e7b-b99d13b1ae8b-kube-api-access-gnwzt\") pod \"e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:33.067986 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:33.067885 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:33.215020 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:33.214989 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s"] Jul 15 12:38:33.217504 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:38:33.217471 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52f9329_318f_454d_8e7b_b99d13b1ae8b.slice/crio-b7b20cee74f99e6993f11bd46506907e92e2b1e89801d2f0a48a776576fbcbd7 WatchSource:0}: Error finding container b7b20cee74f99e6993f11bd46506907e92e2b1e89801d2f0a48a776576fbcbd7: Status 404 returned error can't find the container with id b7b20cee74f99e6993f11bd46506907e92e2b1e89801d2f0a48a776576fbcbd7 Jul 15 12:38:33.220110 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:33.220074 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:38:33.220224 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:33.220144 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:38:33.220224 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:33.220183 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:38:33.543134 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:33.543100 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" event={"ID":"f52f9329-318f-454d-8e7b-b99d13b1ae8b","Type":"ContainerStarted","Data":"4f25bac0e861c4713a1df0cf103859568bdcf60e27c24082aa15d39b6c76ef3b"} Jul 15 12:38:33.543569 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:33.543155 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" event={"ID":"f52f9329-318f-454d-8e7b-b99d13b1ae8b","Type":"ContainerStarted","Data":"b7b20cee74f99e6993f11bd46506907e92e2b1e89801d2f0a48a776576fbcbd7"} Jul 15 12:38:33.881764 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:33.881708 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:34.022261 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:34.022223 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:34.068819 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:34.068791 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:34.090125 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:34.090097 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:34.108872 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:34.108821 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" podStartSLOduration=2.108807277 podStartE2EDuration="2.108807277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:38:33.565876237 +0000 UTC m=+2380.979404590" watchObservedRunningTime="2026-07-15 12:38:34.108807277 +0000 UTC m=+2381.522335627" Jul 15 12:38:34.548197 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:34.548161 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:34.549262 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:34.549240 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:35.493162 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:35.493122 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:35.532838 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:35.532811 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:35.592273 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:35.592230 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:35.679292 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:35.679255 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:35.766694 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:35.766601 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:36.323037 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:36.323005 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:36.424915 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:36.424878 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:40.048125 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:40.048096 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:40.581259 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:40.581178 2570 generic.go:356] "Generic (PLEG): container finished" podID="b059b0db-2cd4-4677-a1ec-909698f2feee" containerID="161503147e00d9ad004638741f6c7a137c546c2450991765d8ec609cd3230034" exitCode=0 Jul 15 12:38:40.581447 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:40.581256 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" event={"ID":"b059b0db-2cd4-4677-a1ec-909698f2feee","Type":"ContainerDied","Data":"161503147e00d9ad004638741f6c7a137c546c2450991765d8ec609cd3230034"} Jul 15 12:38:40.581770 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:40.581721 2570 scope.go:117] "RemoveContainer" containerID="161503147e00d9ad004638741f6c7a137c546c2450991765d8ec609cd3230034" Jul 15 12:38:40.582805 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:40.582775 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:38:40.582879 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:40.582861 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:38:40.582933 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:40.582911 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:38:41.495986 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:41.495942 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:41.588688 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:41.588607 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" event={"ID":"b059b0db-2cd4-4677-a1ec-909698f2feee","Type":"ContainerStarted","Data":"1e55b381cd84d9d8764b429bc7a48b9bdef904c6195aa5bcb42f80d354a9995f"} Jul 15 12:38:42.077045 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:42.077008 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:42.078650 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:42.078291 2570 patch_prober.go:28] interesting pod/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 15 12:38:42.078650 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:42.078365 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" podUID="b059b0db-2cd4-4677-a1ec-909698f2feee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 15 12:38:43.077092 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:43.077048 2570 patch_prober.go:28] interesting pod/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 15 12:38:43.077683 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:43.077647 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" podUID="b059b0db-2cd4-4677-a1ec-909698f2feee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 15 12:38:44.077331 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:44.077281 2570 patch_prober.go:28] interesting pod/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 15 12:38:44.077873 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:44.077352 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" podUID="b059b0db-2cd4-4677-a1ec-909698f2feee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 15 12:38:45.077463 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:45.077411 2570 patch_prober.go:28] interesting pod/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 15 12:38:45.078031 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:45.077500 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" podUID="b059b0db-2cd4-4677-a1ec-909698f2feee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 15 12:38:46.076791 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:46.076725 2570 patch_prober.go:28] interesting pod/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 15 12:38:46.077103 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:46.077072 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" podUID="b059b0db-2cd4-4677-a1ec-909698f2feee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 15 12:38:47.094169 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:47.094139 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:47.094599 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:47.094427 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:47.095315 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:47.095296 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:38:48.696851 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:48.696811 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:48.798330 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:48.798293 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:50.589941 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:50.589893 2570 patch_prober.go:28] interesting pod/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 12:38:50.590347 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:50.589969 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" podUID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 12:38:51.298927 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:51.298896 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:51.398656 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:51.398622 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:52.654617 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:52.654583 2570 generic.go:356] "Generic (PLEG): container finished" podID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" containerID="4f25bac0e861c4713a1df0cf103859568bdcf60e27c24082aa15d39b6c76ef3b" exitCode=0 Jul 15 12:38:52.655090 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:52.654662 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" event={"ID":"f52f9329-318f-454d-8e7b-b99d13b1ae8b","Type":"ContainerDied","Data":"4f25bac0e861c4713a1df0cf103859568bdcf60e27c24082aa15d39b6c76ef3b"} Jul 15 12:38:52.655174 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:52.655157 2570 scope.go:117] "RemoveContainer" containerID="4f25bac0e861c4713a1df0cf103859568bdcf60e27c24082aa15d39b6c76ef3b" Jul 15 12:38:52.656159 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:52.656131 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:38:52.656221 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:52.656206 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:38:52.656257 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:52.656237 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:38:53.291520 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:53.291496 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:38:53.333437 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:53.333407 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:38:53.661927 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:53.661891 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" event={"ID":"f52f9329-318f-454d-8e7b-b99d13b1ae8b","Type":"ContainerStarted","Data":"7b0fc4737bd2f69122de440cde24d448f744776cd0c557da8cc9c74eb07b399e"} Jul 15 12:38:54.069116 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:54.069014 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:54.069297 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:54.069149 2570 patch_prober.go:28] interesting pod/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 15 12:38:54.069297 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:54.069192 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" podUID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 15 12:38:55.069374 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:55.068923 2570 patch_prober.go:28] interesting pod/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 15 12:38:55.069374 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:55.068991 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" podUID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 15 12:38:56.069359 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:56.069315 2570 patch_prober.go:28] interesting pod/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 15 12:38:56.069582 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:56.069387 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" podUID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 15 12:38:57.069134 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:57.069090 2570 patch_prober.go:28] interesting pod/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 15 12:38:57.069452 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:57.069421 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" podUID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 15 12:38:57.801371 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:57.801332 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:58.072658 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.072574 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:58.073023 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.073002 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:58.073696 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.073678 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:38:58.129068 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.129030 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:58.243602 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.243567 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:58.352679 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.352648 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p"] Jul 15 12:38:58.352918 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.352892 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" podUID="57b16b4c-781c-4247-87da-0c0f9064f112" containerName="maas-api" containerID="cri-o://211a0610adbb312d616343629321a37873d469d9da4d05ee57b380450abadfdc" gracePeriod=30 Jul 15 12:38:58.384069 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.384042 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:58.597537 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.597512 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:58.697477 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.697393 2570 generic.go:356] "Generic (PLEG): container finished" podID="57b16b4c-781c-4247-87da-0c0f9064f112" containerID="211a0610adbb312d616343629321a37873d469d9da4d05ee57b380450abadfdc" exitCode=0 Jul 15 12:38:58.697477 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.697439 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" Jul 15 12:38:58.697687 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.697475 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" event={"ID":"57b16b4c-781c-4247-87da-0c0f9064f112","Type":"ContainerDied","Data":"211a0610adbb312d616343629321a37873d469d9da4d05ee57b380450abadfdc"} Jul 15 12:38:58.697687 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.697529 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p" event={"ID":"57b16b4c-781c-4247-87da-0c0f9064f112","Type":"ContainerDied","Data":"57d52f76f77fa51330d460095b3e936608a7fd5d936e69c19262c6d7a462fae7"} Jul 15 12:38:58.697687 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.697560 2570 scope.go:117] "RemoveContainer" containerID="211a0610adbb312d616343629321a37873d469d9da4d05ee57b380450abadfdc" Jul 15 12:38:58.706431 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.706414 2570 scope.go:117] "RemoveContainer" containerID="211a0610adbb312d616343629321a37873d469d9da4d05ee57b380450abadfdc" Jul 15 12:38:58.706683 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:38:58.706668 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211a0610adbb312d616343629321a37873d469d9da4d05ee57b380450abadfdc\": container with ID starting with 211a0610adbb312d616343629321a37873d469d9da4d05ee57b380450abadfdc not found: ID does not exist" containerID="211a0610adbb312d616343629321a37873d469d9da4d05ee57b380450abadfdc" Jul 15 12:38:58.706724 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.706696 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211a0610adbb312d616343629321a37873d469d9da4d05ee57b380450abadfdc"} err="failed to get container status \"211a0610adbb312d616343629321a37873d469d9da4d05ee57b380450abadfdc\": rpc error: code = NotFound desc = could not find container \"211a0610adbb312d616343629321a37873d469d9da4d05ee57b380450abadfdc\": container with ID starting with 211a0610adbb312d616343629321a37873d469d9da4d05ee57b380450abadfdc not found: ID does not exist" Jul 15 12:38:58.755729 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.755700 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b16b4c-781c-4247-87da-0c0f9064f112-openshift-service-ca\") pod \"57b16b4c-781c-4247-87da-0c0f9064f112\" (UID: \"57b16b4c-781c-4247-87da-0c0f9064f112\") " Jul 15 12:38:58.755876 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.755764 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfz8p\" (UniqueName: \"kubernetes.io/projected/57b16b4c-781c-4247-87da-0c0f9064f112-kube-api-access-qfz8p\") pod \"57b16b4c-781c-4247-87da-0c0f9064f112\" (UID: \"57b16b4c-781c-4247-87da-0c0f9064f112\") " Jul 15 12:38:58.755876 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.755791 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b16b4c-781c-4247-87da-0c0f9064f112-maas-api-tls\") pod \"57b16b4c-781c-4247-87da-0c0f9064f112\" (UID: \"57b16b4c-781c-4247-87da-0c0f9064f112\") " Jul 15 12:38:58.756134 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.756109 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b16b4c-781c-4247-87da-0c0f9064f112-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57b16b4c-781c-4247-87da-0c0f9064f112" (UID: "57b16b4c-781c-4247-87da-0c0f9064f112"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:38:58.758073 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.758048 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57b16b4c-781c-4247-87da-0c0f9064f112-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57b16b4c-781c-4247-87da-0c0f9064f112" (UID: "57b16b4c-781c-4247-87da-0c0f9064f112"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:38:58.758169 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.758110 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b16b4c-781c-4247-87da-0c0f9064f112-kube-api-access-qfz8p" (OuterVolumeSpecName: "kube-api-access-qfz8p") pod "57b16b4c-781c-4247-87da-0c0f9064f112" (UID: "57b16b4c-781c-4247-87da-0c0f9064f112"). InnerVolumeSpecName "kube-api-access-qfz8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:38:58.857254 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.857220 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b16b4c-781c-4247-87da-0c0f9064f112-openshift-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:58.857254 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.857251 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfz8p\" (UniqueName: \"kubernetes.io/projected/57b16b4c-781c-4247-87da-0c0f9064f112-kube-api-access-qfz8p\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:58.857686 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:58.857264 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57b16b4c-781c-4247-87da-0c0f9064f112-maas-api-tls\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:38:59.017973 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:59.017938 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p"] Jul 15 12:38:59.022160 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:59.022138 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-858a49-5b5878cbdc-nbf5p"] Jul 15 12:38:59.091833 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:38:59.091804 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57b16b4c-781c-4247-87da-0c0f9064f112" path="/var/lib/kubelet/pods/57b16b4c-781c-4247-87da-0c0f9064f112/volumes" Jul 15 12:39:03.264030 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:03.263991 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:09.345646 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:09.345610 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:09.445985 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:09.445951 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:13.933877 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:13.933799 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k"] Jul 15 12:39:13.934278 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:13.934047 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" podUID="b059b0db-2cd4-4677-a1ec-909698f2feee" containerName="istio-proxy" containerID="cri-o://1e55b381cd84d9d8764b429bc7a48b9bdef904c6195aa5bcb42f80d354a9995f" gracePeriod=30 Jul 15 12:39:13.970068 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:13.970041 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:14.188205 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:14.188113 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:14.289531 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:14.289497 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4"] Jul 15 12:39:14.289925 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:14.289911 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57b16b4c-781c-4247-87da-0c0f9064f112" containerName="maas-api" Jul 15 12:39:14.289980 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:14.289927 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="57b16b4c-781c-4247-87da-0c0f9064f112" containerName="maas-api" Jul 15 12:39:14.290025 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:14.289995 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="57b16b4c-781c-4247-87da-0c0f9064f112" containerName="maas-api" Jul 15 12:39:14.294432 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:14.294413 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4" Jul 15 12:39:14.296300 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:14.296276 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tf4tp\"" Jul 15 12:39:14.301110 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:14.301088 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4"] Jul 15 12:39:14.397908 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:14.397876 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4faf9ca-2f36-44c7-8c10-e539df9d4af8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4\" (UID: \"e4faf9ca-2f36-44c7-8c10-e539df9d4af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4" Jul 15 12:39:14.498855 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:14.498723 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4faf9ca-2f36-44c7-8c10-e539df9d4af8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4\" (UID: \"e4faf9ca-2f36-44c7-8c10-e539df9d4af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4" Jul 15 12:39:14.499384 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:14.499362 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4faf9ca-2f36-44c7-8c10-e539df9d4af8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4\" (UID: \"e4faf9ca-2f36-44c7-8c10-e539df9d4af8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4" Jul 15 12:39:14.607137 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:14.607100 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4" Jul 15 12:39:14.748316 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:14.748245 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4"] Jul 15 12:39:14.750992 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:39:14.750923 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4faf9ca_2f36_44c7_8c10_e539df9d4af8.slice/crio-c6b8d34ca4dd68eee53dedc83fa5ff36a0a3555c28c71916af7baca54fa0cf38 WatchSource:0}: Error finding container c6b8d34ca4dd68eee53dedc83fa5ff36a0a3555c28c71916af7baca54fa0cf38: Status 404 returned error can't find the container with id c6b8d34ca4dd68eee53dedc83fa5ff36a0a3555c28c71916af7baca54fa0cf38 Jul 15 12:39:14.783133 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:14.783104 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4" event={"ID":"e4faf9ca-2f36-44c7-8c10-e539df9d4af8","Type":"ContainerStarted","Data":"c6b8d34ca4dd68eee53dedc83fa5ff36a0a3555c28c71916af7baca54fa0cf38"} Jul 15 12:39:15.789604 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:15.789508 2570 generic.go:356] "Generic (PLEG): container finished" podID="e4faf9ca-2f36-44c7-8c10-e539df9d4af8" containerID="3b529a82cccd1009f35427bdda1fc933f2a0126656ff47adc55cffdeed1be311" exitCode=0 Jul 15 12:39:15.789981 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:15.789596 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4" event={"ID":"e4faf9ca-2f36-44c7-8c10-e539df9d4af8","Type":"ContainerDied","Data":"3b529a82cccd1009f35427bdda1fc933f2a0126656ff47adc55cffdeed1be311"} Jul 15 12:39:16.931126 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:16.931101 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4" Jul 15 12:39:17.020673 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:17.020631 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4faf9ca-2f36-44c7-8c10-e539df9d4af8-maas-api-service-ca\") pod \"e4faf9ca-2f36-44c7-8c10-e539df9d4af8\" (UID: \"e4faf9ca-2f36-44c7-8c10-e539df9d4af8\") " Jul 15 12:39:17.021070 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:17.021045 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4faf9ca-2f36-44c7-8c10-e539df9d4af8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e4faf9ca-2f36-44c7-8c10-e539df9d4af8" (UID: "e4faf9ca-2f36-44c7-8c10-e539df9d4af8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:39:17.096758 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:17.096698 2570 patch_prober.go:28] interesting pod/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 12:39:17.096921 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:17.096789 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" podUID="b059b0db-2cd4-4677-a1ec-909698f2feee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 12:39:17.122153 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:17.122124 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4faf9ca-2f36-44c7-8c10-e539df9d4af8-maas-api-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:17.800693 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:17.800660 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4" Jul 15 12:39:17.800874 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:17.800721 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4" event={"ID":"e4faf9ca-2f36-44c7-8c10-e539df9d4af8","Type":"ContainerDied","Data":"c6b8d34ca4dd68eee53dedc83fa5ff36a0a3555c28c71916af7baca54fa0cf38"} Jul 15 12:39:17.800874 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:17.800797 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6b8d34ca4dd68eee53dedc83fa5ff36a0a3555c28c71916af7baca54fa0cf38" Jul 15 12:39:19.190046 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.190023 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:39:19.244709 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.244614 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgrrk\" (UniqueName: \"kubernetes.io/projected/b059b0db-2cd4-4677-a1ec-909698f2feee-kube-api-access-rgrrk\") pod \"b059b0db-2cd4-4677-a1ec-909698f2feee\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " Jul 15 12:39:19.244709 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.244662 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-token\") pod \"b059b0db-2cd4-4677-a1ec-909698f2feee\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " Jul 15 12:39:19.244709 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.244702 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-data\") pod \"b059b0db-2cd4-4677-a1ec-909698f2feee\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " Jul 15 12:39:19.245038 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.244765 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-credential-socket\") pod \"b059b0db-2cd4-4677-a1ec-909698f2feee\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " Jul 15 12:39:19.245038 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.244795 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b059b0db-2cd4-4677-a1ec-909698f2feee-istiod-ca-cert\") pod \"b059b0db-2cd4-4677-a1ec-909698f2feee\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " Jul 15 12:39:19.245038 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.244837 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-workload-certs\") pod \"b059b0db-2cd4-4677-a1ec-909698f2feee\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " Jul 15 12:39:19.245038 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.244879 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-podinfo\") pod \"b059b0db-2cd4-4677-a1ec-909698f2feee\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " Jul 15 12:39:19.245038 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.244947 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-envoy\") pod \"b059b0db-2cd4-4677-a1ec-909698f2feee\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " Jul 15 12:39:19.245038 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.245001 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-workload-socket\") pod \"b059b0db-2cd4-4677-a1ec-909698f2feee\" (UID: \"b059b0db-2cd4-4677-a1ec-909698f2feee\") " Jul 15 12:39:19.245338 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.245141 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b059b0db-2cd4-4677-a1ec-909698f2feee" (UID: "b059b0db-2cd4-4677-a1ec-909698f2feee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:39:19.245402 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.245368 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b059b0db-2cd4-4677-a1ec-909698f2feee" (UID: "b059b0db-2cd4-4677-a1ec-909698f2feee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:39:19.245402 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.245380 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-credential-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:19.245485 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.245391 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b059b0db-2cd4-4677-a1ec-909698f2feee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b059b0db-2cd4-4677-a1ec-909698f2feee" (UID: "b059b0db-2cd4-4677-a1ec-909698f2feee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:39:19.245609 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.245584 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b059b0db-2cd4-4677-a1ec-909698f2feee" (UID: "b059b0db-2cd4-4677-a1ec-909698f2feee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:39:19.246185 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.246116 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-data" (OuterVolumeSpecName: "istio-data") pod "b059b0db-2cd4-4677-a1ec-909698f2feee" (UID: "b059b0db-2cd4-4677-a1ec-909698f2feee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:39:19.247170 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.247135 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b059b0db-2cd4-4677-a1ec-909698f2feee-kube-api-access-rgrrk" (OuterVolumeSpecName: "kube-api-access-rgrrk") pod "b059b0db-2cd4-4677-a1ec-909698f2feee" (UID: "b059b0db-2cd4-4677-a1ec-909698f2feee"). InnerVolumeSpecName "kube-api-access-rgrrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:39:19.247170 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.247167 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-token" (OuterVolumeSpecName: "istio-token") pod "b059b0db-2cd4-4677-a1ec-909698f2feee" (UID: "b059b0db-2cd4-4677-a1ec-909698f2feee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:39:19.247366 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.247180 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b059b0db-2cd4-4677-a1ec-909698f2feee" (UID: "b059b0db-2cd4-4677-a1ec-909698f2feee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:39:19.247506 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.247487 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b059b0db-2cd4-4677-a1ec-909698f2feee" (UID: "b059b0db-2cd4-4677-a1ec-909698f2feee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:39:19.346703 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.346667 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-data\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:19.346703 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.346697 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b059b0db-2cd4-4677-a1ec-909698f2feee-istiod-ca-cert\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:19.346703 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.346706 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-workload-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:19.346703 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.346714 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-podinfo\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:19.346976 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.346723 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-envoy\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:19.346976 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.346769 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b059b0db-2cd4-4677-a1ec-909698f2feee-workload-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:19.346976 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.346779 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgrrk\" (UniqueName: \"kubernetes.io/projected/b059b0db-2cd4-4677-a1ec-909698f2feee-kube-api-access-rgrrk\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:19.346976 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.346788 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b059b0db-2cd4-4677-a1ec-909698f2feee-istio-token\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:19.814127 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.814094 2570 generic.go:356] "Generic (PLEG): container finished" podID="b059b0db-2cd4-4677-a1ec-909698f2feee" containerID="1e55b381cd84d9d8764b429bc7a48b9bdef904c6195aa5bcb42f80d354a9995f" exitCode=0 Jul 15 12:39:19.814304 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.814157 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" event={"ID":"b059b0db-2cd4-4677-a1ec-909698f2feee","Type":"ContainerDied","Data":"1e55b381cd84d9d8764b429bc7a48b9bdef904c6195aa5bcb42f80d354a9995f"} Jul 15 12:39:19.814304 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.814162 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" Jul 15 12:39:19.814304 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.814231 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k" event={"ID":"b059b0db-2cd4-4677-a1ec-909698f2feee","Type":"ContainerDied","Data":"88785008b82d54343756b00c48d9f14c4beaece193d2dd3b6e02cff1b69c77d9"} Jul 15 12:39:19.814304 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.814256 2570 scope.go:117] "RemoveContainer" containerID="1e55b381cd84d9d8764b429bc7a48b9bdef904c6195aa5bcb42f80d354a9995f" Jul 15 12:39:19.824070 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.824048 2570 scope.go:117] "RemoveContainer" containerID="161503147e00d9ad004638741f6c7a137c546c2450991765d8ec609cd3230034" Jul 15 12:39:19.832195 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.832167 2570 scope.go:117] "RemoveContainer" containerID="1e55b381cd84d9d8764b429bc7a48b9bdef904c6195aa5bcb42f80d354a9995f" Jul 15 12:39:19.832549 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:39:19.832520 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e55b381cd84d9d8764b429bc7a48b9bdef904c6195aa5bcb42f80d354a9995f\": container with ID starting with 1e55b381cd84d9d8764b429bc7a48b9bdef904c6195aa5bcb42f80d354a9995f not found: ID does not exist" containerID="1e55b381cd84d9d8764b429bc7a48b9bdef904c6195aa5bcb42f80d354a9995f" Jul 15 12:39:19.832669 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.832562 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e55b381cd84d9d8764b429bc7a48b9bdef904c6195aa5bcb42f80d354a9995f"} err="failed to get container status \"1e55b381cd84d9d8764b429bc7a48b9bdef904c6195aa5bcb42f80d354a9995f\": rpc error: code = NotFound desc = could not find container \"1e55b381cd84d9d8764b429bc7a48b9bdef904c6195aa5bcb42f80d354a9995f\": container with ID starting with 1e55b381cd84d9d8764b429bc7a48b9bdef904c6195aa5bcb42f80d354a9995f not found: ID does not exist" Jul 15 12:39:19.832669 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.832587 2570 scope.go:117] "RemoveContainer" containerID="161503147e00d9ad004638741f6c7a137c546c2450991765d8ec609cd3230034" Jul 15 12:39:19.833019 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:39:19.832982 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161503147e00d9ad004638741f6c7a137c546c2450991765d8ec609cd3230034\": container with ID starting with 161503147e00d9ad004638741f6c7a137c546c2450991765d8ec609cd3230034 not found: ID does not exist" containerID="161503147e00d9ad004638741f6c7a137c546c2450991765d8ec609cd3230034" Jul 15 12:39:19.833115 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.833017 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161503147e00d9ad004638741f6c7a137c546c2450991765d8ec609cd3230034"} err="failed to get container status \"161503147e00d9ad004638741f6c7a137c546c2450991765d8ec609cd3230034\": rpc error: code = NotFound desc = could not find container \"161503147e00d9ad004638741f6c7a137c546c2450991765d8ec609cd3230034\": container with ID starting with 161503147e00d9ad004638741f6c7a137c546c2450991765d8ec609cd3230034 not found: ID does not exist" Jul 15 12:39:19.835412 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.835391 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k"] Jul 15 12:39:19.838778 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:19.838757 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-858a49-openshift-default-cfcdb75b5-zmb6k"] Jul 15 12:39:21.092269 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:21.092237 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b059b0db-2cd4-4677-a1ec-909698f2feee" path="/var/lib/kubelet/pods/b059b0db-2cd4-4677-a1ec-909698f2feee/volumes" Jul 15 12:39:26.597114 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:26.597065 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:29.398905 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:29.398862 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:35.545548 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:35.545499 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:35.616749 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:35.616713 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:40.086142 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:40.086091 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s"] Jul 15 12:39:40.086544 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:40.086380 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" podUID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" containerName="istio-proxy" containerID="cri-o://7b0fc4737bd2f69122de440cde24d448f744776cd0c557da8cc9c74eb07b399e" gracePeriod=30 Jul 15 12:39:40.119950 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:40.119918 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:40.423673 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:40.423637 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:40.759890 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:40.759801 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:41.042959 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.042871 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5"] Jul 15 12:39:41.043307 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.043291 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4faf9ca-2f36-44c7-8c10-e539df9d4af8" containerName="revoke-keys" Jul 15 12:39:41.043350 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.043310 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4faf9ca-2f36-44c7-8c10-e539df9d4af8" containerName="revoke-keys" Jul 15 12:39:41.043350 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.043338 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b059b0db-2cd4-4677-a1ec-909698f2feee" containerName="istio-proxy" Jul 15 12:39:41.043350 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.043344 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b059b0db-2cd4-4677-a1ec-909698f2feee" containerName="istio-proxy" Jul 15 12:39:41.043442 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.043423 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b059b0db-2cd4-4677-a1ec-909698f2feee" containerName="istio-proxy" Jul 15 12:39:41.043442 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.043434 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b059b0db-2cd4-4677-a1ec-909698f2feee" containerName="istio-proxy" Jul 15 12:39:41.043442 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.043441 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4faf9ca-2f36-44c7-8c10-e539df9d4af8" containerName="revoke-keys" Jul 15 12:39:41.046453 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.046430 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.048526 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.048508 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-a9c7da-openshift-default-dockercfg-v95vm\"" Jul 15 12:39:41.057833 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.057809 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5"] Jul 15 12:39:41.081685 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.081647 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:41.146212 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.146180 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-workload-socket\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.146212 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.146213 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed09391-b89a-4f8b-a96b-1103ed825928-istiod-ca-cert\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.146778 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.146244 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-token\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.146778 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.146300 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-envoy\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.146778 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.146350 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd2c9\" (UniqueName: \"kubernetes.io/projected/0ed09391-b89a-4f8b-a96b-1103ed825928-kube-api-access-sd2c9\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.146778 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.146372 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-data\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.146778 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.146454 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-workload-certs\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.146778 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.146483 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-podinfo\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.146778 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.146507 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-credential-socket\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.238167 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.238132 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:41.248027 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.248004 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-workload-socket\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.248136 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.248035 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed09391-b89a-4f8b-a96b-1103ed825928-istiod-ca-cert\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.248136 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.248067 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-token\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.248136 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.248111 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-envoy\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.248301 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.248136 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd2c9\" (UniqueName: \"kubernetes.io/projected/0ed09391-b89a-4f8b-a96b-1103ed825928-kube-api-access-sd2c9\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.248301 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.248162 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-data\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.248301 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.248218 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-workload-certs\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.248301 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.248242 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-podinfo\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.248301 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.248280 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-credential-socket\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.248548 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.248460 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-workload-socket\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.248617 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.248595 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-workload-certs\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.248704 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.248676 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-data\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.248833 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.248818 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-credential-socket\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.249183 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.249162 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed09391-b89a-4f8b-a96b-1103ed825928-istiod-ca-cert\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.250559 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.250534 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-envoy\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.250918 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.250894 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-podinfo\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.255351 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.255326 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-token\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.255433 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.255364 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd2c9\" (UniqueName: \"kubernetes.io/projected/0ed09391-b89a-4f8b-a96b-1103ed825928-kube-api-access-sd2c9\") pod \"e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.356756 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.356706 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:41.499632 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.499525 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5"] Jul 15 12:39:41.502323 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:39:41.502296 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed09391_b89a_4f8b_a96b_1103ed825928.slice/crio-b3923fc9eaabb639a89eb0b7733c83127c11a4ee0f99be822655e2369d205005 WatchSource:0}: Error finding container b3923fc9eaabb639a89eb0b7733c83127c11a4ee0f99be822655e2369d205005: Status 404 returned error can't find the container with id b3923fc9eaabb639a89eb0b7733c83127c11a4ee0f99be822655e2369d205005 Jul 15 12:39:41.504470 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.504435 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:39:41.504559 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.504511 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:39:41.504559 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.504538 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:39:41.929104 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.929064 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" event={"ID":"0ed09391-b89a-4f8b-a96b-1103ed825928","Type":"ContainerStarted","Data":"560b3b6620fecab0e245a90c26b1d24e87cf2dc636225fccf7368a186f45a2c6"} Jul 15 12:39:41.929291 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.929121 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" event={"ID":"0ed09391-b89a-4f8b-a96b-1103ed825928","Type":"ContainerStarted","Data":"b3923fc9eaabb639a89eb0b7733c83127c11a4ee0f99be822655e2369d205005"} Jul 15 12:39:41.950241 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:41.949532 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" podStartSLOduration=0.949510388 podStartE2EDuration="949.510388ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:39:41.948010722 +0000 UTC m=+2449.361539089" watchObservedRunningTime="2026-07-15 12:39:41.949510388 +0000 UTC m=+2449.363038741" Jul 15 12:39:42.357303 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:42.357274 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:42.361940 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:42.361917 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:42.934805 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:42.934771 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:42.935715 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:42.935693 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:39:43.075334 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:43.075295 2570 patch_prober.go:28] interesting pod/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 12:39:43.075506 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:43.075354 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" podUID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 12:39:43.877749 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:43.877700 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:43.970122 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:43.970090 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:44.040583 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.040552 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6"] Jul 15 12:39:44.041002 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.040987 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b059b0db-2cd4-4677-a1ec-909698f2feee" containerName="istio-proxy" Jul 15 12:39:44.041055 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.041005 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b059b0db-2cd4-4677-a1ec-909698f2feee" containerName="istio-proxy" Jul 15 12:39:44.044425 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.044401 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:39:44.046210 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.046192 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a9c7da-serving-cert\"" Jul 15 12:39:44.058843 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.058818 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6"] Jul 15 12:39:44.073194 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.073161 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381267a1-4d7f-49a3-abd1-5d0880673757-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6\" (UID: \"381267a1-4d7f-49a3-abd1-5d0880673757\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:39:44.073374 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.073289 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381267a1-4d7f-49a3-abd1-5d0880673757-maas-api-tls\") pod \"maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6\" (UID: \"381267a1-4d7f-49a3-abd1-5d0880673757\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:39:44.073374 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.073309 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf9rs\" (UniqueName: \"kubernetes.io/projected/381267a1-4d7f-49a3-abd1-5d0880673757-kube-api-access-zf9rs\") pod \"maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6\" (UID: \"381267a1-4d7f-49a3-abd1-5d0880673757\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:39:44.100168 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.100135 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:44.173858 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.173779 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381267a1-4d7f-49a3-abd1-5d0880673757-maas-api-tls\") pod \"maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6\" (UID: \"381267a1-4d7f-49a3-abd1-5d0880673757\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:39:44.173858 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.173813 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf9rs\" (UniqueName: \"kubernetes.io/projected/381267a1-4d7f-49a3-abd1-5d0880673757-kube-api-access-zf9rs\") pod \"maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6\" (UID: \"381267a1-4d7f-49a3-abd1-5d0880673757\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:39:44.173858 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.173852 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381267a1-4d7f-49a3-abd1-5d0880673757-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6\" (UID: \"381267a1-4d7f-49a3-abd1-5d0880673757\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:39:44.174118 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:39:44.173939 2570 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-serving-cert: secret "maas-api-e2e-shared-a-a9c7da-serving-cert" not found Jul 15 12:39:44.174118 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:39:44.174015 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/381267a1-4d7f-49a3-abd1-5d0880673757-maas-api-tls podName:381267a1-4d7f-49a3-abd1-5d0880673757 nodeName:}" failed. No retries permitted until 2026-07-15 12:39:44.673996301 +0000 UTC m=+2452.087524633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/381267a1-4d7f-49a3-abd1-5d0880673757-maas-api-tls") pod "maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" (UID: "381267a1-4d7f-49a3-abd1-5d0880673757") : secret "maas-api-e2e-shared-a-a9c7da-serving-cert" not found Jul 15 12:39:44.174549 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.174529 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381267a1-4d7f-49a3-abd1-5d0880673757-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6\" (UID: \"381267a1-4d7f-49a3-abd1-5d0880673757\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:39:44.187363 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.184288 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf9rs\" (UniqueName: \"kubernetes.io/projected/381267a1-4d7f-49a3-abd1-5d0880673757-kube-api-access-zf9rs\") pod \"maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6\" (UID: \"381267a1-4d7f-49a3-abd1-5d0880673757\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:39:44.267101 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.267063 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:44.558442 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.558362 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:44.661091 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.661053 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:44.676127 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.676090 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381267a1-4d7f-49a3-abd1-5d0880673757-maas-api-tls\") pod \"maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6\" (UID: \"381267a1-4d7f-49a3-abd1-5d0880673757\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:39:44.678634 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.678604 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381267a1-4d7f-49a3-abd1-5d0880673757-maas-api-tls\") pod \"maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6\" (UID: \"381267a1-4d7f-49a3-abd1-5d0880673757\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:39:44.956052 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:44.956018 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:39:45.098990 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.098964 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6"] Jul 15 12:39:45.101350 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:39:45.101318 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381267a1_4d7f_49a3_abd1_5d0880673757.slice/crio-b3326e98647ae160d8ec2c6b7625e421edddf57e3c74bf7d3089bc05ad67dc92 WatchSource:0}: Error finding container b3326e98647ae160d8ec2c6b7625e421edddf57e3c74bf7d3089bc05ad67dc92: Status 404 returned error can't find the container with id b3326e98647ae160d8ec2c6b7625e421edddf57e3c74bf7d3089bc05ad67dc92 Jul 15 12:39:45.363998 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.363975 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:39:45.381919 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.381896 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-workload-socket\") pod \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " Jul 15 12:39:45.382042 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.381930 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-workload-certs\") pod \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " Jul 15 12:39:45.382042 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.381981 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-podinfo\") pod \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " Jul 15 12:39:45.382042 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.381996 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-envoy\") pod \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " Jul 15 12:39:45.382042 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.382012 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnwzt\" (UniqueName: \"kubernetes.io/projected/f52f9329-318f-454d-8e7b-b99d13b1ae8b-kube-api-access-gnwzt\") pod \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " Jul 15 12:39:45.382265 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.382045 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-token\") pod \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " Jul 15 12:39:45.382265 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.382083 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-data\") pod \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " Jul 15 12:39:45.382265 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.382220 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-credential-socket\") pod \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " Jul 15 12:39:45.382412 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.382276 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istiod-ca-cert\") pod \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\" (UID: \"f52f9329-318f-454d-8e7b-b99d13b1ae8b\") " Jul 15 12:39:45.382412 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.382301 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f52f9329-318f-454d-8e7b-b99d13b1ae8b" (UID: "f52f9329-318f-454d-8e7b-b99d13b1ae8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:39:45.382523 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.382500 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f52f9329-318f-454d-8e7b-b99d13b1ae8b" (UID: "f52f9329-318f-454d-8e7b-b99d13b1ae8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:39:45.382618 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.382571 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f52f9329-318f-454d-8e7b-b99d13b1ae8b" (UID: "f52f9329-318f-454d-8e7b-b99d13b1ae8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:39:45.382678 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.382659 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-credential-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:45.382916 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.382860 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f52f9329-318f-454d-8e7b-b99d13b1ae8b" (UID: "f52f9329-318f-454d-8e7b-b99d13b1ae8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:39:45.382916 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.382878 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-workload-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:45.383285 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.383260 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-data" (OuterVolumeSpecName: "istio-data") pod "f52f9329-318f-454d-8e7b-b99d13b1ae8b" (UID: "f52f9329-318f-454d-8e7b-b99d13b1ae8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:39:45.384707 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.384671 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52f9329-318f-454d-8e7b-b99d13b1ae8b-kube-api-access-gnwzt" (OuterVolumeSpecName: "kube-api-access-gnwzt") pod "f52f9329-318f-454d-8e7b-b99d13b1ae8b" (UID: "f52f9329-318f-454d-8e7b-b99d13b1ae8b"). InnerVolumeSpecName "kube-api-access-gnwzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:39:45.385413 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.385381 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f52f9329-318f-454d-8e7b-b99d13b1ae8b" (UID: "f52f9329-318f-454d-8e7b-b99d13b1ae8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:39:45.385413 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.385388 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f52f9329-318f-454d-8e7b-b99d13b1ae8b" (UID: "f52f9329-318f-454d-8e7b-b99d13b1ae8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:39:45.386081 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.385910 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-token" (OuterVolumeSpecName: "istio-token") pod "f52f9329-318f-454d-8e7b-b99d13b1ae8b" (UID: "f52f9329-318f-454d-8e7b-b99d13b1ae8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:39:45.483556 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.483520 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-workload-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:45.483556 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.483546 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-podinfo\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:45.483556 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.483555 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-envoy\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:45.483556 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.483563 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnwzt\" (UniqueName: \"kubernetes.io/projected/f52f9329-318f-454d-8e7b-b99d13b1ae8b-kube-api-access-gnwzt\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:45.483848 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.483572 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-token\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:45.483848 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.483580 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istio-data\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:45.483848 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.483587 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52f9329-318f-454d-8e7b-b99d13b1ae8b-istiod-ca-cert\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:39:45.952924 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.952889 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" event={"ID":"381267a1-4d7f-49a3-abd1-5d0880673757","Type":"ContainerStarted","Data":"841c78d32ad0c421ab598d6b8849484a9c3e8d4ffc73b5545c7c83807940c5a9"} Jul 15 12:39:45.953114 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.952950 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" event={"ID":"381267a1-4d7f-49a3-abd1-5d0880673757","Type":"ContainerStarted","Data":"b3326e98647ae160d8ec2c6b7625e421edddf57e3c74bf7d3089bc05ad67dc92"} Jul 15 12:39:45.953114 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.953059 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:39:45.954454 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.954433 2570 generic.go:356] "Generic (PLEG): container finished" podID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" containerID="7b0fc4737bd2f69122de440cde24d448f744776cd0c557da8cc9c74eb07b399e" exitCode=0 Jul 15 12:39:45.954506 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.954472 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" Jul 15 12:39:45.954542 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.954503 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" event={"ID":"f52f9329-318f-454d-8e7b-b99d13b1ae8b","Type":"ContainerDied","Data":"7b0fc4737bd2f69122de440cde24d448f744776cd0c557da8cc9c74eb07b399e"} Jul 15 12:39:45.954542 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.954538 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s" event={"ID":"f52f9329-318f-454d-8e7b-b99d13b1ae8b","Type":"ContainerDied","Data":"b7b20cee74f99e6993f11bd46506907e92e2b1e89801d2f0a48a776576fbcbd7"} Jul 15 12:39:45.954605 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.954561 2570 scope.go:117] "RemoveContainer" containerID="7b0fc4737bd2f69122de440cde24d448f744776cd0c557da8cc9c74eb07b399e" Jul 15 12:39:45.964377 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.964180 2570 scope.go:117] "RemoveContainer" containerID="4f25bac0e861c4713a1df0cf103859568bdcf60e27c24082aa15d39b6c76ef3b" Jul 15 12:39:45.971625 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.971556 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" podStartSLOduration=1.44067184 podStartE2EDuration="1.971543074s" totalImagesPullingTime="530.871234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:39:45.969733625 +0000 UTC m=+2453.383261978" watchObservedRunningTime="2026-07-15 12:39:45.971543074 +0000 UTC m=+2453.385071466" Jul 15 12:39:45.972841 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.972825 2570 scope.go:117] "RemoveContainer" containerID="7b0fc4737bd2f69122de440cde24d448f744776cd0c557da8cc9c74eb07b399e" Jul 15 12:39:45.973072 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:39:45.973056 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0fc4737bd2f69122de440cde24d448f744776cd0c557da8cc9c74eb07b399e\": container with ID starting with 7b0fc4737bd2f69122de440cde24d448f744776cd0c557da8cc9c74eb07b399e not found: ID does not exist" containerID="7b0fc4737bd2f69122de440cde24d448f744776cd0c557da8cc9c74eb07b399e" Jul 15 12:39:45.973125 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.973079 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0fc4737bd2f69122de440cde24d448f744776cd0c557da8cc9c74eb07b399e"} err="failed to get container status \"7b0fc4737bd2f69122de440cde24d448f744776cd0c557da8cc9c74eb07b399e\": rpc error: code = NotFound desc = could not find container \"7b0fc4737bd2f69122de440cde24d448f744776cd0c557da8cc9c74eb07b399e\": container with ID starting with 7b0fc4737bd2f69122de440cde24d448f744776cd0c557da8cc9c74eb07b399e not found: ID does not exist" Jul 15 12:39:45.973125 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.973094 2570 scope.go:117] "RemoveContainer" containerID="4f25bac0e861c4713a1df0cf103859568bdcf60e27c24082aa15d39b6c76ef3b" Jul 15 12:39:45.973316 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:39:45.973297 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f25bac0e861c4713a1df0cf103859568bdcf60e27c24082aa15d39b6c76ef3b\": container with ID starting with 4f25bac0e861c4713a1df0cf103859568bdcf60e27c24082aa15d39b6c76ef3b not found: ID does not exist" containerID="4f25bac0e861c4713a1df0cf103859568bdcf60e27c24082aa15d39b6c76ef3b" Jul 15 12:39:45.973370 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.973325 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f25bac0e861c4713a1df0cf103859568bdcf60e27c24082aa15d39b6c76ef3b"} err="failed to get container status \"4f25bac0e861c4713a1df0cf103859568bdcf60e27c24082aa15d39b6c76ef3b\": rpc error: code = NotFound desc = could not find container \"4f25bac0e861c4713a1df0cf103859568bdcf60e27c24082aa15d39b6c76ef3b\": container with ID starting with 4f25bac0e861c4713a1df0cf103859568bdcf60e27c24082aa15d39b6c76ef3b not found: ID does not exist" Jul 15 12:39:45.984817 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.984789 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s"] Jul 15 12:39:45.986688 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:45.986667 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-137ff6-openshift-default-b996fcd8c-55z2s"] Jul 15 12:39:47.092894 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:47.092863 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" path="/var/lib/kubelet/pods/f52f9329-318f-454d-8e7b-b99d13b1ae8b/volumes" Jul 15 12:39:48.398473 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:48.398437 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:48.713553 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:48.713472 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:48.988276 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:48.988200 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5"] Jul 15 12:39:48.988602 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:48.988590 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" containerName="istio-proxy" Jul 15 12:39:48.988646 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:48.988604 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" containerName="istio-proxy" Jul 15 12:39:48.988646 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:48.988630 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" containerName="istio-proxy" Jul 15 12:39:48.988646 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:48.988636 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" containerName="istio-proxy" Jul 15 12:39:48.988757 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:48.988693 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" containerName="istio-proxy" Jul 15 12:39:48.988757 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:48.988702 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f52f9329-318f-454d-8e7b-b99d13b1ae8b" containerName="istio-proxy" Jul 15 12:39:48.993135 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:48.993115 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:48.994983 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:48.994961 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b2e819-openshift-default-dockercfg-s45s7\"" Jul 15 12:39:49.014382 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.014351 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5"] Jul 15 12:39:49.018160 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.018134 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-credential-socket\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.018301 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.018170 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-token\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.018301 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.018198 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-workload-socket\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.018301 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.018248 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-workload-certs\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.018301 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.018278 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-envoy\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.018526 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.018370 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvnxw\" (UniqueName: \"kubernetes.io/projected/4a4d1519-5114-443b-8e6d-c72a258b80a6-kube-api-access-gvnxw\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.018526 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.018401 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a4d1519-5114-443b-8e6d-c72a258b80a6-istiod-ca-cert\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.018526 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.018439 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-podinfo\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.018526 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.018471 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-data\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.020710 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.020687 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:49.119173 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.119132 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-credential-socket\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.119173 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.119176 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-token\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.119388 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.119295 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-workload-socket\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.119388 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.119333 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-workload-certs\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.119388 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.119357 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-envoy\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.119519 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.119452 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvnxw\" (UniqueName: \"kubernetes.io/projected/4a4d1519-5114-443b-8e6d-c72a258b80a6-kube-api-access-gvnxw\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.119519 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.119487 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a4d1519-5114-443b-8e6d-c72a258b80a6-istiod-ca-cert\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.119628 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.119530 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-podinfo\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.119628 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.119580 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-data\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.119731 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.119660 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-workload-socket\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.119731 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.119531 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-credential-socket\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.119731 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.119709 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-workload-certs\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.119904 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.119872 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-data\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.120273 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.120169 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a4d1519-5114-443b-8e6d-c72a258b80a6-istiod-ca-cert\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.121785 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.121761 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-envoy\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.122031 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.122014 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-podinfo\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.126379 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.126356 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvnxw\" (UniqueName: \"kubernetes.io/projected/4a4d1519-5114-443b-8e6d-c72a258b80a6-kube-api-access-gvnxw\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.126471 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.126395 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-token\") pod \"e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.202171 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.202136 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:49.303652 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.303569 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:49.442628 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.442596 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5"] Jul 15 12:39:49.446325 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:39:49.446296 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4d1519_5114_443b_8e6d_c72a258b80a6.slice/crio-eeacdb965aa644549bf343d564faf64c119f47ed22a7911f9b3c0fcc2be8d01b WatchSource:0}: Error finding container eeacdb965aa644549bf343d564faf64c119f47ed22a7911f9b3c0fcc2be8d01b: Status 404 returned error can't find the container with id eeacdb965aa644549bf343d564faf64c119f47ed22a7911f9b3c0fcc2be8d01b Jul 15 12:39:49.448563 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.448528 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:39:49.448651 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.448595 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:39:49.448651 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.448623 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:39:49.982409 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.982369 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" event={"ID":"4a4d1519-5114-443b-8e6d-c72a258b80a6","Type":"ContainerStarted","Data":"62723a6f94d16a5e6a8c4a75bb2043e7ff5dc01c6a195a67e05f6b2ed4153d1b"} Jul 15 12:39:49.982628 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:49.982422 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" event={"ID":"4a4d1519-5114-443b-8e6d-c72a258b80a6","Type":"ContainerStarted","Data":"eeacdb965aa644549bf343d564faf64c119f47ed22a7911f9b3c0fcc2be8d01b"} Jul 15 12:39:50.004202 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:50.004149 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" podStartSLOduration=2.004131324 podStartE2EDuration="2.004131324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:39:50.001418133 +0000 UTC m=+2457.414946502" watchObservedRunningTime="2026-07-15 12:39:50.004131324 +0000 UTC m=+2457.417659674" Jul 15 12:39:50.303854 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:50.303765 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:50.309584 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:50.309560 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:50.987604 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:50.987569 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:50.988580 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:50.988560 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:39:51.937356 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:51.937319 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:51.968156 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:51.968128 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:39:52.040669 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:52.040636 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:52.188287 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:52.188204 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:52.627305 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:52.627268 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:52.728444 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:52.728410 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:56.484334 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:56.484298 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:58.977051 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:58.977000 2570 patch_prober.go:28] interesting pod/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 12:39:58.977917 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:39:58.977695 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" podUID="0ed09391-b89a-4f8b-a96b-1103ed825928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 12:40:00.034189 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:00.034099 2570 generic.go:356] "Generic (PLEG): container finished" podID="0ed09391-b89a-4f8b-a96b-1103ed825928" containerID="560b3b6620fecab0e245a90c26b1d24e87cf2dc636225fccf7368a186f45a2c6" exitCode=0 Jul 15 12:40:00.034189 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:00.034167 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" event={"ID":"0ed09391-b89a-4f8b-a96b-1103ed825928","Type":"ContainerDied","Data":"560b3b6620fecab0e245a90c26b1d24e87cf2dc636225fccf7368a186f45a2c6"} Jul 15 12:40:00.034624 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:00.034608 2570 scope.go:117] "RemoveContainer" containerID="560b3b6620fecab0e245a90c26b1d24e87cf2dc636225fccf7368a186f45a2c6" Jul 15 12:40:00.035601 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:00.035577 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:40:00.035658 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:00.035640 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:40:00.035697 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:00.035669 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:40:01.042055 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:01.042020 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" event={"ID":"0ed09391-b89a-4f8b-a96b-1103ed825928","Type":"ContainerStarted","Data":"f4800286d63bd1c17ec839519b60eb45f98e1c3738322a2524b753d6e12be60a"} Jul 15 12:40:01.357159 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:01.357122 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:40:01.357361 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:01.357314 2570 patch_prober.go:28] interesting pod/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 15 12:40:01.357434 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:01.357387 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" podUID="0ed09391-b89a-4f8b-a96b-1103ed825928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 15 12:40:02.357258 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:02.357222 2570 patch_prober.go:28] interesting pod/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 15 12:40:02.357651 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:02.357282 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" podUID="0ed09391-b89a-4f8b-a96b-1103ed825928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 15 12:40:03.357770 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:03.357720 2570 patch_prober.go:28] interesting pod/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 15 12:40:03.358142 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:03.357793 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" podUID="0ed09391-b89a-4f8b-a96b-1103ed825928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 15 12:40:04.357866 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:04.357815 2570 patch_prober.go:28] interesting pod/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Jul 15 12:40:04.358333 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:04.357893 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" podUID="0ed09391-b89a-4f8b-a96b-1103ed825928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Jul 15 12:40:05.340282 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:05.340245 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:05.360731 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:05.360707 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:40:05.361181 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:05.361154 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:40:05.361803 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:05.361785 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:40:05.439204 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:05.439167 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:06.448331 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:06.447884 2570 patch_prober.go:28] interesting pod/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": read tcp 10.133.0.2:41744->10.133.0.69:15021: read: connection reset by peer" start-of-body= Jul 15 12:40:06.448331 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:06.447959 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": read tcp 10.133.0.2:41744->10.133.0.69:15021: read: connection reset by peer" Jul 15 12:40:07.075899 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:07.075870 2570 generic.go:356] "Generic (PLEG): container finished" podID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerID="62723a6f94d16a5e6a8c4a75bb2043e7ff5dc01c6a195a67e05f6b2ed4153d1b" exitCode=0 Jul 15 12:40:07.076083 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:07.075956 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" event={"ID":"4a4d1519-5114-443b-8e6d-c72a258b80a6","Type":"ContainerDied","Data":"62723a6f94d16a5e6a8c4a75bb2043e7ff5dc01c6a195a67e05f6b2ed4153d1b"} Jul 15 12:40:07.076450 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:07.076436 2570 scope.go:117] "RemoveContainer" containerID="62723a6f94d16a5e6a8c4a75bb2043e7ff5dc01c6a195a67e05f6b2ed4153d1b" Jul 15 12:40:07.077414 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:07.077384 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:40:07.077471 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:07.077458 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:40:07.077506 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:07.077487 2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 15 12:40:08.083538 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:08.083497 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" event={"ID":"4a4d1519-5114-443b-8e6d-c72a258b80a6","Type":"ContainerStarted","Data":"cb1e487b199fe6394dca0874284ef1e04697fcd27b108503340ecd5facee2493"} Jul 15 12:40:08.304165 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:08.304125 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:40:08.304342 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:08.304311 2570 patch_prober.go:28] interesting pod/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 15 12:40:08.304388 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:08.304359 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 15 12:40:09.304792 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:09.304731 2570 patch_prober.go:28] interesting pod/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 15 12:40:09.305206 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:09.304818 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 15 12:40:10.304800 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:10.304730 2570 patch_prober.go:28] interesting pod/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 15 12:40:10.305236 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:10.304830 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 15 12:40:11.304915 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:11.304880 2570 patch_prober.go:28] interesting pod/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 15 12:40:11.305336 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:11.304937 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 15 12:40:12.303889 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:12.303855 2570 patch_prober.go:28] interesting pod/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 15 12:40:12.304154 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:12.303907 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 15 12:40:13.304096 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:13.304058 2570 patch_prober.go:28] interesting pod/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Jul 15 12:40:13.304503 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:13.304129 2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Jul 15 12:40:13.494713 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:13.494672 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:13.595006 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:13.594972 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:14.308545 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:14.308518 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:40:14.309016 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:14.308838 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:40:14.309538 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:14.309518 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:40:22.805182 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:22.805145 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:57.952765 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:57.952646 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4"] Jul 15 12:40:57.957641 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:57.957612 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:57.960553 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:57.960525 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b2e819\"/\"openshift-service-ca.crt\"" Jul 15 12:40:57.960553 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:57.960554 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b2e819\"/\"kube-root-ca.crt\"" Jul 15 12:40:57.960903 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:57.960858 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b2e819\"/\"auth-test-model-b2e819-kserve-self-signed-certs\"" Jul 15 12:40:57.961251 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:57.961161 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b2e819\"/\"default-dockercfg-24brn\"" Jul 15 12:40:57.964685 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:57.964060 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4"] Jul 15 12:40:57.976248 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:57.976189 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:58.021943 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.021908 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0f7a2c-c828-4e61-855d-3f01f8b19921-tls-certs\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.022126 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.021953 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-model-cache\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.022126 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.022023 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwkph\" (UniqueName: \"kubernetes.io/projected/4e0f7a2c-c828-4e61-855d-3f01f8b19921-kube-api-access-mwkph\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.022126 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.022119 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-home\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.022305 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.022195 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-tmp-dir\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.022305 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.022252 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-dshm\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.123717 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.123663 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-home\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.123996 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.123762 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-tmp-dir\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.123996 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.123822 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-dshm\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.124255 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.124170 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-tmp-dir\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.124255 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.123849 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0f7a2c-c828-4e61-855d-3f01f8b19921-tls-certs\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.124255 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.124250 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-home\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.124558 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.124270 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-model-cache\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.124615 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.124559 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-model-cache\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.124615 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.124561 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwkph\" (UniqueName: \"kubernetes.io/projected/4e0f7a2c-c828-4e61-855d-3f01f8b19921-kube-api-access-mwkph\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.126298 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.126279 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-dshm\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.126607 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.126589 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0f7a2c-c828-4e61-855d-3f01f8b19921-tls-certs\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.132060 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.132041 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwkph\" (UniqueName: \"kubernetes.io/projected/4e0f7a2c-c828-4e61-855d-3f01f8b19921-kube-api-access-mwkph\") pod \"auth-test-model-b2e819-kserve-bb9b798ff-qr5b4\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.271005 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.270920 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:40:58.407205 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:58.407175 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4"] Jul 15 12:40:58.408846 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:40:58.408804 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e0f7a2c_c828_4e61_855d_3f01f8b19921.slice/crio-f51018f3e07ed1e91519e2ea05dcd296474a5b883bcf43b18f477ec9662bdc1c WatchSource:0}: Error finding container f51018f3e07ed1e91519e2ea05dcd296474a5b883bcf43b18f477ec9662bdc1c: Status 404 returned error can't find the container with id f51018f3e07ed1e91519e2ea05dcd296474a5b883bcf43b18f477ec9662bdc1c Jul 15 12:40:59.351634 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:59.351597 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" event={"ID":"4e0f7a2c-c828-4e61-855d-3f01f8b19921","Type":"ContainerStarted","Data":"7d660a4b06add025d5a080349fc86c07bce1ca45b9c2b848ed3685defc2e20d2"} Jul 15 12:40:59.352059 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:59.351642 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" event={"ID":"4e0f7a2c-c828-4e61-855d-3f01f8b19921","Type":"ContainerStarted","Data":"f51018f3e07ed1e91519e2ea05dcd296474a5b883bcf43b18f477ec9662bdc1c"} Jul 15 12:40:59.367105 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:40:59.367023 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" podStartSLOduration=2.367006031 podStartE2EDuration="2.367006031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:40:59.366267687 +0000 UTC m=+2526.779796039" watchObservedRunningTime="2026-07-15 12:40:59.367006031 +0000 UTC m=+2526.780534382" Jul 15 12:41:08.272168 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:41:08.272134 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:41:08.272537 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:41:08.272184 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:41:08.284548 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:41:08.284524 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:41:08.410306 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:41:08.410277 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:41:09.083216 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:41:09.083174 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:58.436234 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:41:58.436186 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:58.497810 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:41:58.497779 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:55.220162 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:42:55.220130 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:56.072197 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:42:56.072163 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4"] Jul 15 12:42:57.298594 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:42:57.298542 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:06.141137 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.141099 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt"] Jul 15 12:43:06.143813 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.143791 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.146510 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.146487 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a9c7da\"/\"default-dockercfg-l5mp8\"" Jul 15 12:43:06.146510 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.146500 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a9c7da\"/\"kube-root-ca.crt\"" Jul 15 12:43:06.146728 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.146518 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a9c7da\"/\"openshift-service-ca.crt\"" Jul 15 12:43:06.146728 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.146500 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a9c7da\"/\"sub-test-model-a9c7da-kserve-self-signed-certs\"" Jul 15 12:43:06.156720 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.156692 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt"] Jul 15 12:43:06.175127 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.175077 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:06.280493 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.280455 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-model-cache\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.280493 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.280492 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbcxs\" (UniqueName: \"kubernetes.io/projected/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-kube-api-access-zbcxs\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.280705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.280517 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-home\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.280705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.280539 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-tmp-dir\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.280705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.280562 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-tls-certs\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.280705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.280584 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-dshm\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.381293 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.381258 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-home\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.381293 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.381298 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-tmp-dir\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.381518 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.381316 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-tls-certs\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.381518 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.381342 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-dshm\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.381518 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.381464 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-model-cache\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.381518 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.381489 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbcxs\" (UniqueName: \"kubernetes.io/projected/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-kube-api-access-zbcxs\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.381805 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.381775 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-tmp-dir\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.381920 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.381805 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-home\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.381920 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.381863 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-model-cache\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.383994 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.383970 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-dshm\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.384299 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.384089 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-tls-certs\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.396846 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.396776 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbcxs\" (UniqueName: \"kubernetes.io/projected/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-kube-api-access-zbcxs\") pod \"sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.453761 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.453702 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:06.595050 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:06.595020 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt"] Jul 15 12:43:06.597130 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:43:06.597101 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podceaca2be_6663_4c4c_a7ea_a5aaf4d25172.slice/crio-2a95635b83f759e2ca6c5202f9a5e60d66ec0093ec811b9c27a015eb92226588 WatchSource:0}: Error finding container 2a95635b83f759e2ca6c5202f9a5e60d66ec0093ec811b9c27a015eb92226588: Status 404 returned error can't find the container with id 2a95635b83f759e2ca6c5202f9a5e60d66ec0093ec811b9c27a015eb92226588 Jul 15 12:43:07.035891 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:07.035802 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" event={"ID":"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172","Type":"ContainerStarted","Data":"3ea1f3d5f7f181830fee7f916c1ec10ad73d6fbeb00f8b3cde6c18657939ca68"} Jul 15 12:43:07.035891 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:07.035842 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" event={"ID":"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172","Type":"ContainerStarted","Data":"2a95635b83f759e2ca6c5202f9a5e60d66ec0093ec811b9c27a015eb92226588"} Jul 15 12:43:07.053358 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:07.053308 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" podStartSLOduration=1.053292326 podStartE2EDuration="1.053292326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:43:07.050831009 +0000 UTC m=+2654.464359360" watchObservedRunningTime="2026-07-15 12:43:07.053292326 +0000 UTC m=+2654.466820716" Jul 15 12:43:11.093080 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.093042 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" podUID="4e0f7a2c-c828-4e61-855d-3f01f8b19921" containerName="main" containerID="cri-o://7d660a4b06add025d5a080349fc86c07bce1ca45b9c2b848ed3685defc2e20d2" gracePeriod=45 Jul 15 12:43:11.340629 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.340603 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:43:11.429904 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.429873 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0f7a2c-c828-4e61-855d-3f01f8b19921-tls-certs\") pod \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " Jul 15 12:43:11.430089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.429929 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-home\") pod \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " Jul 15 12:43:11.430089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.429955 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwkph\" (UniqueName: \"kubernetes.io/projected/4e0f7a2c-c828-4e61-855d-3f01f8b19921-kube-api-access-mwkph\") pod \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " Jul 15 12:43:11.430089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.430013 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-model-cache\") pod \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " Jul 15 12:43:11.430089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.430029 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-tmp-dir\") pod \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " Jul 15 12:43:11.430089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.430055 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-dshm\") pod \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\" (UID: \"4e0f7a2c-c828-4e61-855d-3f01f8b19921\") " Jul 15 12:43:11.430350 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.430222 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-home" (OuterVolumeSpecName: "home") pod "4e0f7a2c-c828-4e61-855d-3f01f8b19921" (UID: "4e0f7a2c-c828-4e61-855d-3f01f8b19921"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:43:11.430350 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.430281 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e0f7a2c-c828-4e61-855d-3f01f8b19921" (UID: "4e0f7a2c-c828-4e61-855d-3f01f8b19921"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:43:11.430350 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.430308 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e0f7a2c-c828-4e61-855d-3f01f8b19921" (UID: "4e0f7a2c-c828-4e61-855d-3f01f8b19921"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:43:11.430501 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.430416 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-model-cache\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:43:11.430501 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.430434 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-tmp-dir\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:43:11.430501 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.430443 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-home\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:43:11.432292 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.432267 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-dshm" (OuterVolumeSpecName: "dshm") pod "4e0f7a2c-c828-4e61-855d-3f01f8b19921" (UID: "4e0f7a2c-c828-4e61-855d-3f01f8b19921"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:43:11.432398 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.432342 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e0f7a2c-c828-4e61-855d-3f01f8b19921-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e0f7a2c-c828-4e61-855d-3f01f8b19921" (UID: "4e0f7a2c-c828-4e61-855d-3f01f8b19921"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:43:11.432398 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.432384 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0f7a2c-c828-4e61-855d-3f01f8b19921-kube-api-access-mwkph" (OuterVolumeSpecName: "kube-api-access-mwkph") pod "4e0f7a2c-c828-4e61-855d-3f01f8b19921" (UID: "4e0f7a2c-c828-4e61-855d-3f01f8b19921"). InnerVolumeSpecName "kube-api-access-mwkph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:43:11.531591 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.531554 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0f7a2c-c828-4e61-855d-3f01f8b19921-tls-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:43:11.531591 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.531588 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwkph\" (UniqueName: \"kubernetes.io/projected/4e0f7a2c-c828-4e61-855d-3f01f8b19921-kube-api-access-mwkph\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:43:11.531591 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:11.531600 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e0f7a2c-c828-4e61-855d-3f01f8b19921-dshm\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:43:12.063186 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:12.063156 2570 generic.go:356] "Generic (PLEG): container finished" podID="4e0f7a2c-c828-4e61-855d-3f01f8b19921" containerID="7d660a4b06add025d5a080349fc86c07bce1ca45b9c2b848ed3685defc2e20d2" exitCode=0 Jul 15 12:43:12.063364 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:12.063210 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" Jul 15 12:43:12.063364 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:12.063248 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" event={"ID":"4e0f7a2c-c828-4e61-855d-3f01f8b19921","Type":"ContainerDied","Data":"7d660a4b06add025d5a080349fc86c07bce1ca45b9c2b848ed3685defc2e20d2"} Jul 15 12:43:12.063364 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:12.063298 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4" event={"ID":"4e0f7a2c-c828-4e61-855d-3f01f8b19921","Type":"ContainerDied","Data":"f51018f3e07ed1e91519e2ea05dcd296474a5b883bcf43b18f477ec9662bdc1c"} Jul 15 12:43:12.063364 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:12.063321 2570 scope.go:117] "RemoveContainer" containerID="7d660a4b06add025d5a080349fc86c07bce1ca45b9c2b848ed3685defc2e20d2" Jul 15 12:43:12.072721 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:12.072703 2570 scope.go:117] "RemoveContainer" containerID="7d660a4b06add025d5a080349fc86c07bce1ca45b9c2b848ed3685defc2e20d2" Jul 15 12:43:12.073022 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:43:12.073004 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d660a4b06add025d5a080349fc86c07bce1ca45b9c2b848ed3685defc2e20d2\": container with ID starting with 7d660a4b06add025d5a080349fc86c07bce1ca45b9c2b848ed3685defc2e20d2 not found: ID does not exist" containerID="7d660a4b06add025d5a080349fc86c07bce1ca45b9c2b848ed3685defc2e20d2" Jul 15 12:43:12.073082 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:12.073032 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d660a4b06add025d5a080349fc86c07bce1ca45b9c2b848ed3685defc2e20d2"} err="failed to get container status \"7d660a4b06add025d5a080349fc86c07bce1ca45b9c2b848ed3685defc2e20d2\": rpc error: code = NotFound desc = could not find container \"7d660a4b06add025d5a080349fc86c07bce1ca45b9c2b848ed3685defc2e20d2\": container with ID starting with 7d660a4b06add025d5a080349fc86c07bce1ca45b9c2b848ed3685defc2e20d2 not found: ID does not exist" Jul 15 12:43:12.084386 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:12.084359 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4"] Jul 15 12:43:12.089398 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:12.089370 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b2e819/auth-test-model-b2e819-kserve-bb9b798ff-qr5b4"] Jul 15 12:43:13.093286 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:13.093251 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e0f7a2c-c828-4e61-855d-3f01f8b19921" path="/var/lib/kubelet/pods/4e0f7a2c-c828-4e61-855d-3f01f8b19921/volumes" Jul 15 12:43:16.454848 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:16.454807 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:16.455303 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:16.454866 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:16.467505 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:16.467483 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:17.101412 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:17.101387 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:43:21.775650 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:21.775610 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:40.646335 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:40.646292 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:52.020518 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:52.020439 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:52.121089 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:52.121045 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:53.322239 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:53.322215 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:43:53.368813 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:53.368790 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7kxg9_52d7772a-574c-4960-9541-e138399719f3/ovn-acl-logging/0.log" Jul 15 12:43:56.720645 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:56.720605 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt"] Jul 15 12:43:56.739545 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:56.739511 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:57.700374 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:43:57.700336 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:10.351483 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.351449 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4"] Jul 15 12:44:10.351895 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.351881 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0f7a2c-c828-4e61-855d-3f01f8b19921" containerName="main" Jul 15 12:44:10.351940 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.351898 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0f7a2c-c828-4e61-855d-3f01f8b19921" containerName="main" Jul 15 12:44:10.351996 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.351984 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0f7a2c-c828-4e61-855d-3f01f8b19921" containerName="main" Jul 15 12:44:10.355369 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.355353 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.357204 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.357187 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a9c7da\"/\"rate-test-model-a9c7da-kserve-self-signed-certs\"" Jul 15 12:44:10.363270 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.363248 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4"] Jul 15 12:44:10.368927 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.368906 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-tls-certs\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.369031 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.368951 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-home\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.369031 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.369017 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sqmg\" (UniqueName: \"kubernetes.io/projected/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-kube-api-access-2sqmg\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.369156 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.369047 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-dshm\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.369156 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.369063 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-tmp-dir\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.369265 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.369176 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-model-cache\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.379580 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.379549 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:10.469990 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.469949 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sqmg\" (UniqueName: \"kubernetes.io/projected/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-kube-api-access-2sqmg\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.470194 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.470000 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-dshm\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.470194 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.470026 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-tmp-dir\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.470194 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.470095 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-model-cache\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.470194 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.470155 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-tls-certs\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.470418 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.470213 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-home\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.470581 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.470556 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-tmp-dir\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.470627 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.470591 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-model-cache\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.470675 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.470633 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-home\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.472443 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.472421 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-dshm\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.472708 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.472688 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-tls-certs\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.477655 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.477635 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sqmg\" (UniqueName: \"kubernetes.io/projected/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-kube-api-access-2sqmg\") pod \"rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.665934 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.665843 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:10.800765 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:10.800720 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4"] Jul 15 12:44:10.802926 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:44:10.802893 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ed4838b_ab71_4faa_8ef6_1f7beecc20a5.slice/crio-06288d2280d6de63294a9a84dc9937ea1c6871d7d36f7a9d1b98f7383fc0ba43 WatchSource:0}: Error finding container 06288d2280d6de63294a9a84dc9937ea1c6871d7d36f7a9d1b98f7383fc0ba43: Status 404 returned error can't find the container with id 06288d2280d6de63294a9a84dc9937ea1c6871d7d36f7a9d1b98f7383fc0ba43 Jul 15 12:44:11.329214 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:11.329176 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:11.380663 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:11.380626 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" event={"ID":"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5","Type":"ContainerStarted","Data":"22b60901968fccebcf6895860ec100124e5cb4947a4586074ae0a6b4fe5e0561"} Jul 15 12:44:11.381059 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:11.380672 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" event={"ID":"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5","Type":"ContainerStarted","Data":"06288d2280d6de63294a9a84dc9937ea1c6871d7d36f7a9d1b98f7383fc0ba43"} Jul 15 12:44:11.396379 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:11.396328 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" podStartSLOduration=1.396310718 podStartE2EDuration="1.396310718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:44:11.395186746 +0000 UTC m=+2718.808715089" watchObservedRunningTime="2026-07-15 12:44:11.396310718 +0000 UTC m=+2718.809839069" Jul 15 12:44:11.741798 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:11.741725 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" podUID="ceaca2be-6663-4c4c-a7ea-a5aaf4d25172" containerName="main" containerID="cri-o://3ea1f3d5f7f181830fee7f916c1ec10ad73d6fbeb00f8b3cde6c18657939ca68" gracePeriod=45 Jul 15 12:44:12.002896 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.002831 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:44:12.085844 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.085804 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-model-cache\") pod \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " Jul 15 12:44:12.086036 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.085856 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-dshm\") pod \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " Jul 15 12:44:12.086036 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.085898 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-tmp-dir\") pod \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " Jul 15 12:44:12.086036 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.085939 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-tls-certs\") pod \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " Jul 15 12:44:12.086036 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.085996 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-home\") pod \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " Jul 15 12:44:12.086036 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.086022 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbcxs\" (UniqueName: \"kubernetes.io/projected/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-kube-api-access-zbcxs\") pod \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\" (UID: \"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172\") " Jul 15 12:44:12.086290 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.086131 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-model-cache" (OuterVolumeSpecName: "model-cache") pod "ceaca2be-6663-4c4c-a7ea-a5aaf4d25172" (UID: "ceaca2be-6663-4c4c-a7ea-a5aaf4d25172"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:44:12.086290 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.086197 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ceaca2be-6663-4c4c-a7ea-a5aaf4d25172" (UID: "ceaca2be-6663-4c4c-a7ea-a5aaf4d25172"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:44:12.086404 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.086294 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-home" (OuterVolumeSpecName: "home") pod "ceaca2be-6663-4c4c-a7ea-a5aaf4d25172" (UID: "ceaca2be-6663-4c4c-a7ea-a5aaf4d25172"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:44:12.086404 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.086352 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-tmp-dir\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:44:12.086404 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.086374 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-model-cache\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:44:12.088415 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.088390 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-dshm" (OuterVolumeSpecName: "dshm") pod "ceaca2be-6663-4c4c-a7ea-a5aaf4d25172" (UID: "ceaca2be-6663-4c4c-a7ea-a5aaf4d25172"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:44:12.088415 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.088394 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-kube-api-access-zbcxs" (OuterVolumeSpecName: "kube-api-access-zbcxs") pod "ceaca2be-6663-4c4c-a7ea-a5aaf4d25172" (UID: "ceaca2be-6663-4c4c-a7ea-a5aaf4d25172"). InnerVolumeSpecName "kube-api-access-zbcxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:44:12.088578 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.088441 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ceaca2be-6663-4c4c-a7ea-a5aaf4d25172" (UID: "ceaca2be-6663-4c4c-a7ea-a5aaf4d25172"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:44:12.187926 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.187888 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-tls-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:44:12.187926 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.187920 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-home\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:44:12.188133 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.187932 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbcxs\" (UniqueName: \"kubernetes.io/projected/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-kube-api-access-zbcxs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:44:12.188133 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.187946 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172-dshm\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:44:12.388226 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.388193 2570 generic.go:356] "Generic (PLEG): container finished" podID="ceaca2be-6663-4c4c-a7ea-a5aaf4d25172" containerID="3ea1f3d5f7f181830fee7f916c1ec10ad73d6fbeb00f8b3cde6c18657939ca68" exitCode=0 Jul 15 12:44:12.388690 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.388240 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" Jul 15 12:44:12.388690 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.388276 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" event={"ID":"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172","Type":"ContainerDied","Data":"3ea1f3d5f7f181830fee7f916c1ec10ad73d6fbeb00f8b3cde6c18657939ca68"} Jul 15 12:44:12.388690 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.388310 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt" event={"ID":"ceaca2be-6663-4c4c-a7ea-a5aaf4d25172","Type":"ContainerDied","Data":"2a95635b83f759e2ca6c5202f9a5e60d66ec0093ec811b9c27a015eb92226588"} Jul 15 12:44:12.388690 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.388328 2570 scope.go:117] "RemoveContainer" containerID="3ea1f3d5f7f181830fee7f916c1ec10ad73d6fbeb00f8b3cde6c18657939ca68" Jul 15 12:44:12.397536 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.397510 2570 scope.go:117] "RemoveContainer" containerID="3ea1f3d5f7f181830fee7f916c1ec10ad73d6fbeb00f8b3cde6c18657939ca68" Jul 15 12:44:12.397830 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:44:12.397806 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea1f3d5f7f181830fee7f916c1ec10ad73d6fbeb00f8b3cde6c18657939ca68\": container with ID starting with 3ea1f3d5f7f181830fee7f916c1ec10ad73d6fbeb00f8b3cde6c18657939ca68 not found: ID does not exist" containerID="3ea1f3d5f7f181830fee7f916c1ec10ad73d6fbeb00f8b3cde6c18657939ca68" Jul 15 12:44:12.397918 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.397841 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea1f3d5f7f181830fee7f916c1ec10ad73d6fbeb00f8b3cde6c18657939ca68"} err="failed to get container status \"3ea1f3d5f7f181830fee7f916c1ec10ad73d6fbeb00f8b3cde6c18657939ca68\": rpc error: code = NotFound desc = could not find container \"3ea1f3d5f7f181830fee7f916c1ec10ad73d6fbeb00f8b3cde6c18657939ca68\": container with ID starting with 3ea1f3d5f7f181830fee7f916c1ec10ad73d6fbeb00f8b3cde6c18657939ca68 not found: ID does not exist" Jul 15 12:44:12.408070 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.408042 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt"] Jul 15 12:44:12.411901 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:12.411878 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a9c7da/sub-test-model-a9c7da-kserve-d4c78cffc-s5kjt"] Jul 15 12:44:13.093023 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:13.092985 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceaca2be-6663-4c4c-a7ea-a5aaf4d25172" path="/var/lib/kubelet/pods/ceaca2be-6663-4c4c-a7ea-a5aaf4d25172/volumes" Jul 15 12:44:20.666605 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:20.666563 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:20.667059 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:20.666698 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:20.679380 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:20.679350 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:44:21.456820 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:44:21.456781 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:45:00.326459 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:45:00.326423 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:06.062290 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:45:06.062251 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:06.162815 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:45:06.162783 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:14.238520 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:45:14.238447 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:14.336230 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:45:14.336192 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:47.032141 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:45:47.032104 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4"] Jul 15 12:45:47.079653 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:45:47.079615 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:48.016611 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:45:48.016576 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:02.053141 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.053089 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" podUID="1ed4838b-ab71-4faa-8ef6-1f7beecc20a5" containerName="main" containerID="cri-o://22b60901968fccebcf6895860ec100124e5cb4947a4586074ae0a6b4fe5e0561" gracePeriod=45 Jul 15 12:46:02.306422 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.306365 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:46:02.444126 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.444091 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-tmp-dir\") pod \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " Jul 15 12:46:02.444126 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.444133 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-dshm\") pod \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " Jul 15 12:46:02.444370 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.444186 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-tls-certs\") pod \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " Jul 15 12:46:02.444370 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.444262 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-model-cache\") pod \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " Jul 15 12:46:02.444370 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.444320 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sqmg\" (UniqueName: \"kubernetes.io/projected/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-kube-api-access-2sqmg\") pod \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " Jul 15 12:46:02.444532 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.444376 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-home\") pod \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\" (UID: \"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5\") " Jul 15 12:46:02.444532 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.444374 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ed4838b-ab71-4faa-8ef6-1f7beecc20a5" (UID: "1ed4838b-ab71-4faa-8ef6-1f7beecc20a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:46:02.444642 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.444558 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ed4838b-ab71-4faa-8ef6-1f7beecc20a5" (UID: "1ed4838b-ab71-4faa-8ef6-1f7beecc20a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:46:02.444703 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.444652 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-home" (OuterVolumeSpecName: "home") pod "1ed4838b-ab71-4faa-8ef6-1f7beecc20a5" (UID: "1ed4838b-ab71-4faa-8ef6-1f7beecc20a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:46:02.444794 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.444714 2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-model-cache\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:02.444794 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.444732 2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-home\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:02.444794 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.444778 2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-tmp-dir\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:02.446588 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.446558 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ed4838b-ab71-4faa-8ef6-1f7beecc20a5" (UID: "1ed4838b-ab71-4faa-8ef6-1f7beecc20a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:46:02.446588 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.446570 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-dshm" (OuterVolumeSpecName: "dshm") pod "1ed4838b-ab71-4faa-8ef6-1f7beecc20a5" (UID: "1ed4838b-ab71-4faa-8ef6-1f7beecc20a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:46:02.446779 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.446587 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-kube-api-access-2sqmg" (OuterVolumeSpecName: "kube-api-access-2sqmg") pod "1ed4838b-ab71-4faa-8ef6-1f7beecc20a5" (UID: "1ed4838b-ab71-4faa-8ef6-1f7beecc20a5"). InnerVolumeSpecName "kube-api-access-2sqmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:46:02.546116 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.546077 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sqmg\" (UniqueName: \"kubernetes.io/projected/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-kube-api-access-2sqmg\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:02.546116 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.546108 2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-dshm\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:02.546116 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.546121 2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5-tls-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:02.983163 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.983133 2570 generic.go:356] "Generic (PLEG): container finished" podID="1ed4838b-ab71-4faa-8ef6-1f7beecc20a5" containerID="22b60901968fccebcf6895860ec100124e5cb4947a4586074ae0a6b4fe5e0561" exitCode=0 Jul 15 12:46:02.983330 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.983175 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" Jul 15 12:46:02.983330 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.983212 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" event={"ID":"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5","Type":"ContainerDied","Data":"22b60901968fccebcf6895860ec100124e5cb4947a4586074ae0a6b4fe5e0561"} Jul 15 12:46:02.983330 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.983261 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4" event={"ID":"1ed4838b-ab71-4faa-8ef6-1f7beecc20a5","Type":"ContainerDied","Data":"06288d2280d6de63294a9a84dc9937ea1c6871d7d36f7a9d1b98f7383fc0ba43"} Jul 15 12:46:02.983330 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.983283 2570 scope.go:117] "RemoveContainer" containerID="22b60901968fccebcf6895860ec100124e5cb4947a4586074ae0a6b4fe5e0561" Jul 15 12:46:02.999897 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:02.999874 2570 scope.go:117] "RemoveContainer" containerID="22b60901968fccebcf6895860ec100124e5cb4947a4586074ae0a6b4fe5e0561" Jul 15 12:46:03.000156 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:46:03.000135 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b60901968fccebcf6895860ec100124e5cb4947a4586074ae0a6b4fe5e0561\": container with ID starting with 22b60901968fccebcf6895860ec100124e5cb4947a4586074ae0a6b4fe5e0561 not found: ID does not exist" containerID="22b60901968fccebcf6895860ec100124e5cb4947a4586074ae0a6b4fe5e0561" Jul 15 12:46:03.000209 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:03.000166 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b60901968fccebcf6895860ec100124e5cb4947a4586074ae0a6b4fe5e0561"} err="failed to get container status \"22b60901968fccebcf6895860ec100124e5cb4947a4586074ae0a6b4fe5e0561\": rpc error: code = NotFound desc = could not find container \"22b60901968fccebcf6895860ec100124e5cb4947a4586074ae0a6b4fe5e0561\": container with ID starting with 22b60901968fccebcf6895860ec100124e5cb4947a4586074ae0a6b4fe5e0561 not found: ID does not exist" Jul 15 12:46:03.006675 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:03.006652 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4"] Jul 15 12:46:03.010675 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:03.010653 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a9c7da/rate-test-model-a9c7da-kserve-5dd6656f9-2xkz4"] Jul 15 12:46:03.092256 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:03.092221 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ed4838b-ab71-4faa-8ef6-1f7beecc20a5" path="/var/lib/kubelet/pods/1ed4838b-ab71-4faa-8ef6-1f7beecc20a5/volumes" Jul 15 12:46:07.076798 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.076750 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb"] Jul 15 12:46:07.077364 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.077343 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ed4838b-ab71-4faa-8ef6-1f7beecc20a5" containerName="main" Jul 15 12:46:07.077448 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.077367 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ed4838b-ab71-4faa-8ef6-1f7beecc20a5" containerName="main" Jul 15 12:46:07.077448 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.077421 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceaca2be-6663-4c4c-a7ea-a5aaf4d25172" containerName="main" Jul 15 12:46:07.077448 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.077431 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceaca2be-6663-4c4c-a7ea-a5aaf4d25172" containerName="main" Jul 15 12:46:07.077611 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.077530 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceaca2be-6663-4c4c-a7ea-a5aaf4d25172" containerName="main" Jul 15 12:46:07.077611 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.077545 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ed4838b-ab71-4faa-8ef6-1f7beecc20a5" containerName="main" Jul 15 12:46:07.082125 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.082100 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb" Jul 15 12:46:07.083976 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.083946 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tf4tp\"" Jul 15 12:46:07.084981 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.084958 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e269861d-ab22-41c1-b885-74ace07d809b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb\" (UID: \"e269861d-ab22-41c1-b885-74ace07d809b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb" Jul 15 12:46:07.096417 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.096388 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb"] Jul 15 12:46:07.185595 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.185558 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e269861d-ab22-41c1-b885-74ace07d809b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb\" (UID: \"e269861d-ab22-41c1-b885-74ace07d809b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb" Jul 15 12:46:07.186249 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.186227 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e269861d-ab22-41c1-b885-74ace07d809b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb\" (UID: \"e269861d-ab22-41c1-b885-74ace07d809b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb" Jul 15 12:46:07.393705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.393672 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb" Jul 15 12:46:07.536563 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.536539 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb"] Jul 15 12:46:07.538429 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:46:07.538403 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode269861d_ab22_41c1_b885_74ace07d809b.slice/crio-4efe094c1da693a87022ad11d75d91cbce866514ce9ad326ff3d5cd15cb89f6f WatchSource:0}: Error finding container 4efe094c1da693a87022ad11d75d91cbce866514ce9ad326ff3d5cd15cb89f6f: Status 404 returned error can't find the container with id 4efe094c1da693a87022ad11d75d91cbce866514ce9ad326ff3d5cd15cb89f6f Jul 15 12:46:07.600821 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:07.600789 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:08.012296 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:08.012261 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb" event={"ID":"e269861d-ab22-41c1-b885-74ace07d809b","Type":"ContainerStarted","Data":"1a5174d970bf29688fb7227dbad4acf008eeb00ebfd75e2394a52e975089deac"} Jul 15 12:46:08.012481 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:08.012332 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb" event={"ID":"e269861d-ab22-41c1-b885-74ace07d809b","Type":"ContainerStarted","Data":"4efe094c1da693a87022ad11d75d91cbce866514ce9ad326ff3d5cd15cb89f6f"} Jul 15 12:46:08.028209 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:08.028159 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb" podStartSLOduration=1.028145534 podStartE2EDuration="1.028145534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:46:08.026358384 +0000 UTC m=+2835.439886746" watchObservedRunningTime="2026-07-15 12:46:08.028145534 +0000 UTC m=+2835.441673896" Jul 15 12:46:08.986561 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:08.986525 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:09.018713 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:09.018672 2570 generic.go:356] "Generic (PLEG): container finished" podID="e269861d-ab22-41c1-b885-74ace07d809b" containerID="1a5174d970bf29688fb7227dbad4acf008eeb00ebfd75e2394a52e975089deac" exitCode=0 Jul 15 12:46:09.018879 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:09.018771 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb" event={"ID":"e269861d-ab22-41c1-b885-74ace07d809b","Type":"ContainerDied","Data":"1a5174d970bf29688fb7227dbad4acf008eeb00ebfd75e2394a52e975089deac"} Jul 15 12:46:10.163557 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:10.163531 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb" Jul 15 12:46:10.214879 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:10.214824 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e269861d-ab22-41c1-b885-74ace07d809b-maas-api-service-ca\") pod \"e269861d-ab22-41c1-b885-74ace07d809b\" (UID: \"e269861d-ab22-41c1-b885-74ace07d809b\") " Jul 15 12:46:10.215195 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:10.215173 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e269861d-ab22-41c1-b885-74ace07d809b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e269861d-ab22-41c1-b885-74ace07d809b" (UID: "e269861d-ab22-41c1-b885-74ace07d809b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:46:10.215768 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:10.215721 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e269861d-ab22-41c1-b885-74ace07d809b-maas-api-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:11.031127 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:11.031097 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb" Jul 15 12:46:11.031305 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:11.031096 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb" event={"ID":"e269861d-ab22-41c1-b885-74ace07d809b","Type":"ContainerDied","Data":"4efe094c1da693a87022ad11d75d91cbce866514ce9ad326ff3d5cd15cb89f6f"} Jul 15 12:46:11.031305 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:11.031255 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4efe094c1da693a87022ad11d75d91cbce866514ce9ad326ff3d5cd15cb89f6f" Jul 15 12:46:17.489664 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:17.489630 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6"] Jul 15 12:46:17.490202 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:17.489879 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" podUID="381267a1-4d7f-49a3-abd1-5d0880673757" containerName="maas-api" containerID="cri-o://841c78d32ad0c421ab598d6b8849484a9c3e8d4ffc73b5545c7c83807940c5a9" gracePeriod=30 Jul 15 12:46:17.551149 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:17.551107 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:17.942791 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:17.942760 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:46:17.986032 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:17.986001 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381267a1-4d7f-49a3-abd1-5d0880673757-openshift-service-ca\") pod \"381267a1-4d7f-49a3-abd1-5d0880673757\" (UID: \"381267a1-4d7f-49a3-abd1-5d0880673757\") " Jul 15 12:46:17.986203 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:17.986133 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381267a1-4d7f-49a3-abd1-5d0880673757-maas-api-tls\") pod \"381267a1-4d7f-49a3-abd1-5d0880673757\" (UID: \"381267a1-4d7f-49a3-abd1-5d0880673757\") " Jul 15 12:46:17.986203 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:17.986173 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf9rs\" (UniqueName: \"kubernetes.io/projected/381267a1-4d7f-49a3-abd1-5d0880673757-kube-api-access-zf9rs\") pod \"381267a1-4d7f-49a3-abd1-5d0880673757\" (UID: \"381267a1-4d7f-49a3-abd1-5d0880673757\") " Jul 15 12:46:17.986481 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:17.986450 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381267a1-4d7f-49a3-abd1-5d0880673757-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "381267a1-4d7f-49a3-abd1-5d0880673757" (UID: "381267a1-4d7f-49a3-abd1-5d0880673757"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:46:17.988405 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:17.988381 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381267a1-4d7f-49a3-abd1-5d0880673757-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "381267a1-4d7f-49a3-abd1-5d0880673757" (UID: "381267a1-4d7f-49a3-abd1-5d0880673757"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:46:17.988405 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:17.988389 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381267a1-4d7f-49a3-abd1-5d0880673757-kube-api-access-zf9rs" (OuterVolumeSpecName: "kube-api-access-zf9rs") pod "381267a1-4d7f-49a3-abd1-5d0880673757" (UID: "381267a1-4d7f-49a3-abd1-5d0880673757"). InnerVolumeSpecName "kube-api-access-zf9rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:46:18.073593 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:18.073513 2570 generic.go:356] "Generic (PLEG): container finished" podID="381267a1-4d7f-49a3-abd1-5d0880673757" containerID="841c78d32ad0c421ab598d6b8849484a9c3e8d4ffc73b5545c7c83807940c5a9" exitCode=0 Jul 15 12:46:18.073593 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:18.073563 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" Jul 15 12:46:18.073824 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:18.073600 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" event={"ID":"381267a1-4d7f-49a3-abd1-5d0880673757","Type":"ContainerDied","Data":"841c78d32ad0c421ab598d6b8849484a9c3e8d4ffc73b5545c7c83807940c5a9"} Jul 15 12:46:18.073824 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:18.073652 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6" event={"ID":"381267a1-4d7f-49a3-abd1-5d0880673757","Type":"ContainerDied","Data":"b3326e98647ae160d8ec2c6b7625e421edddf57e3c74bf7d3089bc05ad67dc92"} Jul 15 12:46:18.073824 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:18.073674 2570 scope.go:117] "RemoveContainer" containerID="841c78d32ad0c421ab598d6b8849484a9c3e8d4ffc73b5545c7c83807940c5a9" Jul 15 12:46:18.083162 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:18.083142 2570 scope.go:117] "RemoveContainer" containerID="841c78d32ad0c421ab598d6b8849484a9c3e8d4ffc73b5545c7c83807940c5a9" Jul 15 12:46:18.083430 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:46:18.083410 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841c78d32ad0c421ab598d6b8849484a9c3e8d4ffc73b5545c7c83807940c5a9\": container with ID starting with 841c78d32ad0c421ab598d6b8849484a9c3e8d4ffc73b5545c7c83807940c5a9 not found: ID does not exist" containerID="841c78d32ad0c421ab598d6b8849484a9c3e8d4ffc73b5545c7c83807940c5a9" Jul 15 12:46:18.083498 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:18.083439 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841c78d32ad0c421ab598d6b8849484a9c3e8d4ffc73b5545c7c83807940c5a9"} err="failed to get container status \"841c78d32ad0c421ab598d6b8849484a9c3e8d4ffc73b5545c7c83807940c5a9\": rpc error: code = NotFound desc = could not find container \"841c78d32ad0c421ab598d6b8849484a9c3e8d4ffc73b5545c7c83807940c5a9\": container with ID starting with 841c78d32ad0c421ab598d6b8849484a9c3e8d4ffc73b5545c7c83807940c5a9 not found: ID does not exist" Jul 15 12:46:18.087435 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:18.087417 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf9rs\" (UniqueName: \"kubernetes.io/projected/381267a1-4d7f-49a3-abd1-5d0880673757-kube-api-access-zf9rs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:18.087498 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:18.087438 2570 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381267a1-4d7f-49a3-abd1-5d0880673757-openshift-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:18.087498 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:18.087448 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381267a1-4d7f-49a3-abd1-5d0880673757-maas-api-tls\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:18.093719 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:18.093694 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6"] Jul 15 12:46:18.096567 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:18.096545 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a9c7da-7cc579798-5g6h6"] Jul 15 12:46:19.092453 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:19.092415 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="381267a1-4d7f-49a3-abd1-5d0880673757" path="/var/lib/kubelet/pods/381267a1-4d7f-49a3-abd1-5d0880673757/volumes" Jul 15 12:46:22.217803 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:22.217763 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:29.014075 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:29.014041 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:32.894675 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:32.894640 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5"] Jul 15 12:46:32.895091 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:32.894886 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" podUID="0ed09391-b89a-4f8b-a96b-1103ed825928" containerName="istio-proxy" containerID="cri-o://f4800286d63bd1c17ec839519b60eb45f98e1c3738322a2524b753d6e12be60a" gracePeriod=30 Jul 15 12:46:32.928310 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:32.928272 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:33.158705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.158610 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:33.264616 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.264581 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z"] Jul 15 12:46:33.264997 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.264983 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e269861d-ab22-41c1-b885-74ace07d809b" containerName="revoke-keys" Jul 15 12:46:33.265102 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.264999 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="e269861d-ab22-41c1-b885-74ace07d809b" containerName="revoke-keys" Jul 15 12:46:33.265102 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.265006 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381267a1-4d7f-49a3-abd1-5d0880673757" containerName="maas-api" Jul 15 12:46:33.265102 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.265012 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="381267a1-4d7f-49a3-abd1-5d0880673757" containerName="maas-api" Jul 15 12:46:33.265102 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.265071 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="e269861d-ab22-41c1-b885-74ace07d809b" containerName="revoke-keys" Jul 15 12:46:33.265102 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.265083 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="381267a1-4d7f-49a3-abd1-5d0880673757" containerName="maas-api" Jul 15 12:46:33.267907 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.267891 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z" Jul 15 12:46:33.269850 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.269830 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tf4tp\"" Jul 15 12:46:33.279112 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.279085 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z"] Jul 15 12:46:33.328267 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.328232 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0b17cd-e929-4b7c-be75-f152934f73bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z\" (UID: \"fb0b17cd-e929-4b7c-be75-f152934f73bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z" Jul 15 12:46:33.429337 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.429245 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0b17cd-e929-4b7c-be75-f152934f73bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z\" (UID: \"fb0b17cd-e929-4b7c-be75-f152934f73bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z" Jul 15 12:46:33.429914 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.429895 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0b17cd-e929-4b7c-be75-f152934f73bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z\" (UID: \"fb0b17cd-e929-4b7c-be75-f152934f73bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z" Jul 15 12:46:33.579336 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.579301 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z" Jul 15 12:46:33.718125 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.718084 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z"] Jul 15 12:46:33.720559 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:46:33.720534 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0b17cd_e929_4b7c_be75_f152934f73bc.slice/crio-e74da490ad2491ea0e4b438c26ef1e38ee70ac2e7ceba426fe942cf03a7cdc27 WatchSource:0}: Error finding container e74da490ad2491ea0e4b438c26ef1e38ee70ac2e7ceba426fe942cf03a7cdc27: Status 404 returned error can't find the container with id e74da490ad2491ea0e4b438c26ef1e38ee70ac2e7ceba426fe942cf03a7cdc27 Jul 15 12:46:33.782605 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.782574 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:33.880480 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:33.880448 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:34.155530 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:34.155497 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z" event={"ID":"fb0b17cd-e929-4b7c-be75-f152934f73bc","Type":"ContainerStarted","Data":"1aa85bfee0c124cd484206e6dd376e06af543f4a5b58628897858df7bce26d54"} Jul 15 12:46:34.155975 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:34.155558 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z" event={"ID":"fb0b17cd-e929-4b7c-be75-f152934f73bc","Type":"ContainerStarted","Data":"e74da490ad2491ea0e4b438c26ef1e38ee70ac2e7ceba426fe942cf03a7cdc27"} Jul 15 12:46:34.181309 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:34.181262 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z" podStartSLOduration=1.181246756 podStartE2EDuration="1.181246756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:46:34.178548477 +0000 UTC m=+2861.592076830" watchObservedRunningTime="2026-07-15 12:46:34.181246756 +0000 UTC m=+2861.594775107" Jul 15 12:46:35.161995 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:35.161957 2570 generic.go:356] "Generic (PLEG): container finished" podID="fb0b17cd-e929-4b7c-be75-f152934f73bc" containerID="1aa85bfee0c124cd484206e6dd376e06af543f4a5b58628897858df7bce26d54" exitCode=0 Jul 15 12:46:35.162408 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:35.162007 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z" event={"ID":"fb0b17cd-e929-4b7c-be75-f152934f73bc","Type":"ContainerDied","Data":"1aa85bfee0c124cd484206e6dd376e06af543f4a5b58628897858df7bce26d54"} Jul 15 12:46:35.363882 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:35.363846 2570 patch_prober.go:28] interesting pod/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 12:46:35.364064 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:35.363898 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" podUID="0ed09391-b89a-4f8b-a96b-1103ed825928" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 12:46:36.301209 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:36.301183 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z" Jul 15 12:46:36.458185 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:36.458096 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0b17cd-e929-4b7c-be75-f152934f73bc-maas-api-service-ca\") pod \"fb0b17cd-e929-4b7c-be75-f152934f73bc\" (UID: \"fb0b17cd-e929-4b7c-be75-f152934f73bc\") " Jul 15 12:46:36.458533 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:36.458509 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0b17cd-e929-4b7c-be75-f152934f73bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fb0b17cd-e929-4b7c-be75-f152934f73bc" (UID: "fb0b17cd-e929-4b7c-be75-f152934f73bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:46:36.559452 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:36.559417 2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb0b17cd-e929-4b7c-be75-f152934f73bc-maas-api-service-ca\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:37.174561 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:37.174529 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z" Jul 15 12:46:37.174758 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:37.174530 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z" event={"ID":"fb0b17cd-e929-4b7c-be75-f152934f73bc","Type":"ContainerDied","Data":"e74da490ad2491ea0e4b438c26ef1e38ee70ac2e7ceba426fe942cf03a7cdc27"} Jul 15 12:46:37.174758 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:37.174656 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e74da490ad2491ea0e4b438c26ef1e38ee70ac2e7ceba426fe942cf03a7cdc27" Jul 15 12:46:38.146536 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.146513 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:46:38.181945 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.181852 2570 generic.go:356] "Generic (PLEG): container finished" podID="0ed09391-b89a-4f8b-a96b-1103ed825928" containerID="f4800286d63bd1c17ec839519b60eb45f98e1c3738322a2524b753d6e12be60a" exitCode=0 Jul 15 12:46:38.181945 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.181908 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" Jul 15 12:46:38.181945 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.181940 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" event={"ID":"0ed09391-b89a-4f8b-a96b-1103ed825928","Type":"ContainerDied","Data":"f4800286d63bd1c17ec839519b60eb45f98e1c3738322a2524b753d6e12be60a"} Jul 15 12:46:38.182179 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.181987 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5" event={"ID":"0ed09391-b89a-4f8b-a96b-1103ed825928","Type":"ContainerDied","Data":"b3923fc9eaabb639a89eb0b7733c83127c11a4ee0f99be822655e2369d205005"} Jul 15 12:46:38.182179 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.182009 2570 scope.go:117] "RemoveContainer" containerID="f4800286d63bd1c17ec839519b60eb45f98e1c3738322a2524b753d6e12be60a" Jul 15 12:46:38.190957 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.190936 2570 scope.go:117] "RemoveContainer" containerID="560b3b6620fecab0e245a90c26b1d24e87cf2dc636225fccf7368a186f45a2c6" Jul 15 12:46:38.199119 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.199100 2570 scope.go:117] "RemoveContainer" containerID="f4800286d63bd1c17ec839519b60eb45f98e1c3738322a2524b753d6e12be60a" Jul 15 12:46:38.199408 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:46:38.199382 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4800286d63bd1c17ec839519b60eb45f98e1c3738322a2524b753d6e12be60a\": container with ID starting with f4800286d63bd1c17ec839519b60eb45f98e1c3738322a2524b753d6e12be60a not found: ID does not exist" containerID="f4800286d63bd1c17ec839519b60eb45f98e1c3738322a2524b753d6e12be60a" Jul 15 12:46:38.199475 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.199423 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4800286d63bd1c17ec839519b60eb45f98e1c3738322a2524b753d6e12be60a"} err="failed to get container status \"f4800286d63bd1c17ec839519b60eb45f98e1c3738322a2524b753d6e12be60a\": rpc error: code = NotFound desc = could not find container \"f4800286d63bd1c17ec839519b60eb45f98e1c3738322a2524b753d6e12be60a\": container with ID starting with f4800286d63bd1c17ec839519b60eb45f98e1c3738322a2524b753d6e12be60a not found: ID does not exist" Jul 15 12:46:38.199475 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.199450 2570 scope.go:117] "RemoveContainer" containerID="560b3b6620fecab0e245a90c26b1d24e87cf2dc636225fccf7368a186f45a2c6" Jul 15 12:46:38.199717 ip-10-0-131-53 kubenswrapper[2570]: E0715 12:46:38.199699 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560b3b6620fecab0e245a90c26b1d24e87cf2dc636225fccf7368a186f45a2c6\": container with ID starting with 560b3b6620fecab0e245a90c26b1d24e87cf2dc636225fccf7368a186f45a2c6 not found: ID does not exist" containerID="560b3b6620fecab0e245a90c26b1d24e87cf2dc636225fccf7368a186f45a2c6" Jul 15 12:46:38.199790 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.199726 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560b3b6620fecab0e245a90c26b1d24e87cf2dc636225fccf7368a186f45a2c6"} err="failed to get container status \"560b3b6620fecab0e245a90c26b1d24e87cf2dc636225fccf7368a186f45a2c6\": rpc error: code = NotFound desc = could not find container \"560b3b6620fecab0e245a90c26b1d24e87cf2dc636225fccf7368a186f45a2c6\": container with ID starting with 560b3b6620fecab0e245a90c26b1d24e87cf2dc636225fccf7368a186f45a2c6 not found: ID does not exist" Jul 15 12:46:38.277069 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277035 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-credential-socket\") pod \"0ed09391-b89a-4f8b-a96b-1103ed825928\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " Jul 15 12:46:38.277220 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277094 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-podinfo\") pod \"0ed09391-b89a-4f8b-a96b-1103ed825928\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " Jul 15 12:46:38.277220 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277120 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-envoy\") pod \"0ed09391-b89a-4f8b-a96b-1103ed825928\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " Jul 15 12:46:38.277220 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277193 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-workload-certs\") pod \"0ed09391-b89a-4f8b-a96b-1103ed825928\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " Jul 15 12:46:38.277404 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277276 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-token\") pod \"0ed09391-b89a-4f8b-a96b-1103ed825928\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " Jul 15 12:46:38.277404 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277310 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd2c9\" (UniqueName: \"kubernetes.io/projected/0ed09391-b89a-4f8b-a96b-1103ed825928-kube-api-access-sd2c9\") pod \"0ed09391-b89a-4f8b-a96b-1103ed825928\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " Jul 15 12:46:38.277404 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277332 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0ed09391-b89a-4f8b-a96b-1103ed825928" (UID: "0ed09391-b89a-4f8b-a96b-1103ed825928"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:46:38.277404 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277343 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed09391-b89a-4f8b-a96b-1103ed825928-istiod-ca-cert\") pod \"0ed09391-b89a-4f8b-a96b-1103ed825928\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " Jul 15 12:46:38.277404 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277402 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-data\") pod \"0ed09391-b89a-4f8b-a96b-1103ed825928\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " Jul 15 12:46:38.277665 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277439 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-workload-socket\") pod \"0ed09391-b89a-4f8b-a96b-1103ed825928\" (UID: \"0ed09391-b89a-4f8b-a96b-1103ed825928\") " Jul 15 12:46:38.277665 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277581 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0ed09391-b89a-4f8b-a96b-1103ed825928" (UID: "0ed09391-b89a-4f8b-a96b-1103ed825928"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:46:38.277665 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277658 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed09391-b89a-4f8b-a96b-1103ed825928-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0ed09391-b89a-4f8b-a96b-1103ed825928" (UID: "0ed09391-b89a-4f8b-a96b-1103ed825928"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:46:38.278047 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277896 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0ed09391-b89a-4f8b-a96b-1103ed825928" (UID: "0ed09391-b89a-4f8b-a96b-1103ed825928"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:46:38.278047 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277924 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed09391-b89a-4f8b-a96b-1103ed825928-istiod-ca-cert\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:38.278047 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277943 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-credential-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:38.278047 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.277956 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-workload-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:38.278688 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.278659 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-data" (OuterVolumeSpecName: "istio-data") pod "0ed09391-b89a-4f8b-a96b-1103ed825928" (UID: "0ed09391-b89a-4f8b-a96b-1103ed825928"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:46:38.279705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.279668 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed09391-b89a-4f8b-a96b-1103ed825928-kube-api-access-sd2c9" (OuterVolumeSpecName: "kube-api-access-sd2c9") pod "0ed09391-b89a-4f8b-a96b-1103ed825928" (UID: "0ed09391-b89a-4f8b-a96b-1103ed825928"). InnerVolumeSpecName "kube-api-access-sd2c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:46:38.279705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.279671 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-token" (OuterVolumeSpecName: "istio-token") pod "0ed09391-b89a-4f8b-a96b-1103ed825928" (UID: "0ed09391-b89a-4f8b-a96b-1103ed825928"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:46:38.279705 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.279670 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0ed09391-b89a-4f8b-a96b-1103ed825928" (UID: "0ed09391-b89a-4f8b-a96b-1103ed825928"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:46:38.279934 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.279812 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0ed09391-b89a-4f8b-a96b-1103ed825928" (UID: "0ed09391-b89a-4f8b-a96b-1103ed825928"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:46:38.379027 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.378982 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd2c9\" (UniqueName: \"kubernetes.io/projected/0ed09391-b89a-4f8b-a96b-1103ed825928-kube-api-access-sd2c9\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:38.379027 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.379020 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-data\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:38.379027 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.379029 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-workload-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:38.379027 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.379038 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-podinfo\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:38.379304 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.379048 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-envoy\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:38.379304 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.379056 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed09391-b89a-4f8b-a96b-1103ed825928-istio-token\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:46:38.502186 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.502152 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5"] Jul 15 12:46:38.507877 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:38.507849 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-a9c7da-openshift-default-64dc9d68ff-nn4v5"] Jul 15 12:46:39.092977 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:39.092948 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ed09391-b89a-4f8b-a96b-1103ed825928" path="/var/lib/kubelet/pods/0ed09391-b89a-4f8b-a96b-1103ed825928/volumes" Jul 15 12:46:43.846914 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:43.846828 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:47.472558 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:47.472520 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:47.571913 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:47.571877 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:48.373464 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:48.373421 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:54.713043 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:54.713011 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:54.823831 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:54.823783 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:59.065652 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:59.065619 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5"] Jul 15 12:46:59.066119 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:59.065859 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerName="istio-proxy" containerID="cri-o://cb1e487b199fe6394dca0874284ef1e04697fcd27b108503340ecd5facee2493" gracePeriod=30 Jul 15 12:46:59.242546 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:59.242498 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:59.311728 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:59.311693 2570 patch_prober.go:28] interesting pod/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 12:46:59.311901 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:59.311766 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 12:46:59.416790 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:59.416730 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:59.792664 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:46:59.792556 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f894584f6-vnc5l_6d613f76-23d4-4392-a2e2-f8658440a072/authorino/0.log" Jul 15 12:47:04.318645 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.318612 2570 generic.go:356] "Generic (PLEG): container finished" podID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerID="cb1e487b199fe6394dca0874284ef1e04697fcd27b108503340ecd5facee2493" exitCode=0 Jul 15 12:47:04.319090 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.318688 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" event={"ID":"4a4d1519-5114-443b-8e6d-c72a258b80a6","Type":"ContainerDied","Data":"cb1e487b199fe6394dca0874284ef1e04697fcd27b108503340ecd5facee2493"} Jul 15 12:47:04.319090 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.318768 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" event={"ID":"4a4d1519-5114-443b-8e6d-c72a258b80a6","Type":"ContainerDied","Data":"eeacdb965aa644549bf343d564faf64c119f47ed22a7911f9b3c0fcc2be8d01b"} Jul 15 12:47:04.319090 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.318787 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeacdb965aa644549bf343d564faf64c119f47ed22a7911f9b3c0fcc2be8d01b" Jul 15 12:47:04.319090 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.318801 2570 scope.go:117] "RemoveContainer" containerID="62723a6f94d16a5e6a8c4a75bb2043e7ff5dc01c6a195a67e05f6b2ed4153d1b" Jul 15 12:47:04.327556 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.327526 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:47:04.414325 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.414298 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-podinfo\") pod \"4a4d1519-5114-443b-8e6d-c72a258b80a6\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " Jul 15 12:47:04.414529 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.414348 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-token\") pod \"4a4d1519-5114-443b-8e6d-c72a258b80a6\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " Jul 15 12:47:04.414529 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.414366 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvnxw\" (UniqueName: \"kubernetes.io/projected/4a4d1519-5114-443b-8e6d-c72a258b80a6-kube-api-access-gvnxw\") pod \"4a4d1519-5114-443b-8e6d-c72a258b80a6\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " Jul 15 12:47:04.414663 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.414558 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-credential-socket\") pod \"4a4d1519-5114-443b-8e6d-c72a258b80a6\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " Jul 15 12:47:04.414663 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.414614 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-workload-socket\") pod \"4a4d1519-5114-443b-8e6d-c72a258b80a6\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " Jul 15 12:47:04.414663 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.414643 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-envoy\") pod \"4a4d1519-5114-443b-8e6d-c72a258b80a6\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " Jul 15 12:47:04.414857 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.414678 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a4d1519-5114-443b-8e6d-c72a258b80a6-istiod-ca-cert\") pod \"4a4d1519-5114-443b-8e6d-c72a258b80a6\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " Jul 15 12:47:04.414857 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.414719 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-workload-certs\") pod \"4a4d1519-5114-443b-8e6d-c72a258b80a6\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " Jul 15 12:47:04.414857 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.414781 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-data\") pod \"4a4d1519-5114-443b-8e6d-c72a258b80a6\" (UID: \"4a4d1519-5114-443b-8e6d-c72a258b80a6\") " Jul 15 12:47:04.415016 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.414936 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4a4d1519-5114-443b-8e6d-c72a258b80a6" (UID: "4a4d1519-5114-443b-8e6d-c72a258b80a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:47:04.415114 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.415025 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a4d1519-5114-443b-8e6d-c72a258b80a6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4a4d1519-5114-443b-8e6d-c72a258b80a6" (UID: "4a4d1519-5114-443b-8e6d-c72a258b80a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:47:04.415114 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.415119 2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-workload-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:47:04.415114 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.415134 2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a4d1519-5114-443b-8e6d-c72a258b80a6-istiod-ca-cert\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:47:04.415405 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.415251 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4a4d1519-5114-443b-8e6d-c72a258b80a6" (UID: "4a4d1519-5114-443b-8e6d-c72a258b80a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:47:04.415405 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.415297 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4a4d1519-5114-443b-8e6d-c72a258b80a6" (UID: "4a4d1519-5114-443b-8e6d-c72a258b80a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:47:04.415866 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.415841 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-data" (OuterVolumeSpecName: "istio-data") pod "4a4d1519-5114-443b-8e6d-c72a258b80a6" (UID: "4a4d1519-5114-443b-8e6d-c72a258b80a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:47:04.416761 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.416718 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-token" (OuterVolumeSpecName: "istio-token") pod "4a4d1519-5114-443b-8e6d-c72a258b80a6" (UID: "4a4d1519-5114-443b-8e6d-c72a258b80a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:47:04.416918 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.416894 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4d1519-5114-443b-8e6d-c72a258b80a6-kube-api-access-gvnxw" (OuterVolumeSpecName: "kube-api-access-gvnxw") pod "4a4d1519-5114-443b-8e6d-c72a258b80a6" (UID: "4a4d1519-5114-443b-8e6d-c72a258b80a6"). InnerVolumeSpecName "kube-api-access-gvnxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:47:04.417065 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.417049 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4a4d1519-5114-443b-8e6d-c72a258b80a6" (UID: "4a4d1519-5114-443b-8e6d-c72a258b80a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:47:04.417220 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.417198 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4a4d1519-5114-443b-8e6d-c72a258b80a6" (UID: "4a4d1519-5114-443b-8e6d-c72a258b80a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:47:04.516240 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.516207 2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-envoy\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:47:04.516240 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.516233 2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-workload-certs\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:47:04.516240 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.516243 2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-data\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:47:04.516473 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.516251 2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-podinfo\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:47:04.516473 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.516259 2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a4d1519-5114-443b-8e6d-c72a258b80a6-istio-token\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:47:04.516473 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.516266 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvnxw\" (UniqueName: \"kubernetes.io/projected/4a4d1519-5114-443b-8e6d-c72a258b80a6-kube-api-access-gvnxw\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:47:04.516473 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:04.516276 2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4d1519-5114-443b-8e6d-c72a258b80a6-credential-socket\") on node \"ip-10-0-131-53.ec2.internal\" DevicePath \"\"" Jul 15 12:47:05.324711 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:05.324678 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5" Jul 15 12:47:05.364862 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:05.364779 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5"] Jul 15 12:47:05.370212 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:05.370185 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b2e819-openshift-default-9dbff47c5-rb7p5"] Jul 15 12:47:05.610981 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:05.610948 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55594846d9-q2qsp_4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02/manager/0.log" Jul 15 12:47:05.959346 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:05.959316 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694c4fcbc8-6vv9f_69ea5e3a-62a0-476b-80de-32af2b08ff93/copy-manifests/0.log" Jul 15 12:47:05.965859 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:05.965833 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694c4fcbc8-6vv9f_69ea5e3a-62a0-476b-80de-32af2b08ff93/manager/0.log" Jul 15 12:47:06.601177 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:06.601142 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-66c7d954f5-qfs4v_e0cc2ed9-7abc-4bcf-8433-63a3891dd6fa/maas-api/0.log" Jul 15 12:47:06.707812 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:06.707783 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735295-b9xql_53c52611-e70f-4c8e-b5b0-6f497ff66c0c/cleanup/0.log" Jul 15 12:47:06.816661 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:06.816629 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735310-r9kxg_d8f0d292-b419-4b2f-81de-7c25e974a297/cleanup/0.log" Jul 15 12:47:07.035799 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:07.035719 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-548eb552-8fjqr_7b5621d2-a481-43cf-a0cb-a787b2f9bc7d/revoke-keys/0.log" Jul 15 12:47:07.092585 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:07.092556 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" path="/var/lib/kubelet/pods/4a4d1519-5114-443b-8e6d-c72a258b80a6/volumes" Jul 15 12:47:07.147596 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:07.147573 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-6f432845-7td4p_b487df91-72d9-4526-b4c3-965aee774a8c/revoke-keys/0.log" Jul 15 12:47:07.371944 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:07.371901 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-137ff6-tzdd4_e4faf9ca-2f36-44c7-8c10-e539df9d4af8/revoke-keys/0.log" Jul 15 12:47:07.483843 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:07.483819 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-99db84ea-ssm4w_6ca53125-90e8-42a4-9a8e-f808b0ba425d/revoke-keys/0.log" Jul 15 12:47:07.705655 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:07.705575 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-6dba6e-z28zt_b74f1452-8689-4e60-8075-b6ab99ce5d62/revoke-keys/0.log" Jul 15 12:47:07.817502 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:07.817474 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-bbac7fd2-64xqj_5cfe7d47-396c-47ab-a37e-139ac9069afd/revoke-keys/0.log" Jul 15 12:47:08.049948 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:08.049870 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-d3e5d67b-7szp4_28edefef-bbd5-4399-91e3-9f917d2bda52/revoke-keys/0.log" Jul 15 12:47:08.161999 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:08.161969 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-a9c7da-ndfdb_e269861d-ab22-41c1-b885-74ace07d809b/revoke-keys/0.log" Jul 15 12:47:08.174543 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:08.174498 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:08.274218 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:08.274181 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-b2e819-vfk9z_fb0b17cd-e929-4b7c-be75-f152934f73bc/revoke-keys/0.log" Jul 15 12:47:08.386829 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:08.386798 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-5q2pj_f572fcfc-1218-42bf-bc41-68056a69f4b5/postgres/0.log" Jul 15 12:47:09.699961 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:09.699932 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f894584f6-vnc5l_6d613f76-23d4-4392-a2e2-f8658440a072/authorino/0.log" Jul 15 12:47:10.043115 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:10.043042 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zgfdl_d95a5f8a-9a7c-414b-a526-6db1b266c278/kuadrant-console-plugin/0.log" Jul 15 12:47:11.269000 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:11.268959 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d7f886b65-v8fmz_621592bf-b388-465a-843a-2c803e0c6c1c/kube-auth-proxy/0.log" Jul 15 12:47:11.775160 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:11.775127 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c864b74c8-z6ccx_f8699bdf-f732-4921-a592-483bea1c84fb/router/0.log" Jul 15 12:47:12.126718 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:12.126690 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-cb4ng_4d5d866e-fb8c-4a7c-8dfc-fa5982ab5100/main/0.log" Jul 15 12:47:18.189939 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.189905 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d79rw/must-gather-5x8xm"] Jul 15 12:47:18.190361 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.190252 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ed09391-b89a-4f8b-a96b-1103ed825928" containerName="istio-proxy" Jul 15 12:47:18.190361 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.190263 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ed09391-b89a-4f8b-a96b-1103ed825928" containerName="istio-proxy" Jul 15 12:47:18.190361 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.190274 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerName="istio-proxy" Jul 15 12:47:18.190361 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.190280 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerName="istio-proxy" Jul 15 12:47:18.190361 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.190299 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0b17cd-e929-4b7c-be75-f152934f73bc" containerName="revoke-keys" Jul 15 12:47:18.190361 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.190304 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0b17cd-e929-4b7c-be75-f152934f73bc" containerName="revoke-keys" Jul 15 12:47:18.190361 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.190312 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerName="istio-proxy" Jul 15 12:47:18.190361 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.190317 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerName="istio-proxy" Jul 15 12:47:18.190619 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.190373 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerName="istio-proxy" Jul 15 12:47:18.190619 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.190381 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0b17cd-e929-4b7c-be75-f152934f73bc" containerName="revoke-keys" Jul 15 12:47:18.190619 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.190389 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a4d1519-5114-443b-8e6d-c72a258b80a6" containerName="istio-proxy" Jul 15 12:47:18.190619 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.190395 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ed09391-b89a-4f8b-a96b-1103ed825928" containerName="istio-proxy" Jul 15 12:47:18.190619 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.190403 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ed09391-b89a-4f8b-a96b-1103ed825928" containerName="istio-proxy" Jul 15 12:47:18.190619 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.190466 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ed09391-b89a-4f8b-a96b-1103ed825928" containerName="istio-proxy" Jul 15 12:47:18.190619 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.190471 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ed09391-b89a-4f8b-a96b-1103ed825928" containerName="istio-proxy" Jul 15 12:47:18.193908 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.193885 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d79rw/must-gather-5x8xm" Jul 15 12:47:18.195684 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.195661 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d79rw\"/\"openshift-service-ca.crt\"" Jul 15 12:47:18.195831 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.195688 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d79rw\"/\"default-dockercfg-l7gjh\"" Jul 15 12:47:18.196124 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.196105 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d79rw\"/\"kube-root-ca.crt\"" Jul 15 12:47:18.205083 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.205056 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d79rw/must-gather-5x8xm"] Jul 15 12:47:18.222464 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.222429 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:18.340556 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.340520 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9343e87-ff89-4e66-88a8-1bb0463dcb36-must-gather-output\") pod \"must-gather-5x8xm\" (UID: \"e9343e87-ff89-4e66-88a8-1bb0463dcb36\") " pod="openshift-must-gather-d79rw/must-gather-5x8xm" Jul 15 12:47:18.340733 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.340565 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr95f\" (UniqueName: \"kubernetes.io/projected/e9343e87-ff89-4e66-88a8-1bb0463dcb36-kube-api-access-sr95f\") pod \"must-gather-5x8xm\" (UID: \"e9343e87-ff89-4e66-88a8-1bb0463dcb36\") " pod="openshift-must-gather-d79rw/must-gather-5x8xm" Jul 15 12:47:18.382789 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.382710 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:18.441506 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.441417 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr95f\" (UniqueName: \"kubernetes.io/projected/e9343e87-ff89-4e66-88a8-1bb0463dcb36-kube-api-access-sr95f\") pod \"must-gather-5x8xm\" (UID: \"e9343e87-ff89-4e66-88a8-1bb0463dcb36\") " pod="openshift-must-gather-d79rw/must-gather-5x8xm" Jul 15 12:47:18.441660 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.441559 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9343e87-ff89-4e66-88a8-1bb0463dcb36-must-gather-output\") pod \"must-gather-5x8xm\" (UID: \"e9343e87-ff89-4e66-88a8-1bb0463dcb36\") " pod="openshift-must-gather-d79rw/must-gather-5x8xm" Jul 15 12:47:18.441942 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.441924 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e9343e87-ff89-4e66-88a8-1bb0463dcb36-must-gather-output\") pod \"must-gather-5x8xm\" (UID: \"e9343e87-ff89-4e66-88a8-1bb0463dcb36\") " pod="openshift-must-gather-d79rw/must-gather-5x8xm" Jul 15 12:47:18.449291 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.449272 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr95f\" (UniqueName: \"kubernetes.io/projected/e9343e87-ff89-4e66-88a8-1bb0463dcb36-kube-api-access-sr95f\") pod \"must-gather-5x8xm\" (UID: \"e9343e87-ff89-4e66-88a8-1bb0463dcb36\") " pod="openshift-must-gather-d79rw/must-gather-5x8xm" Jul 15 12:47:18.502956 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.502930 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d79rw/must-gather-5x8xm" Jul 15 12:47:18.637793 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.637770 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d79rw/must-gather-5x8xm"] Jul 15 12:47:18.639976 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:47:18.639951 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9343e87_ff89_4e66_88a8_1bb0463dcb36.slice/crio-54317b9f414e40296f04cb452761cd3a598453ef3d57a6e861412f6616944e54 WatchSource:0}: Error finding container 54317b9f414e40296f04cb452761cd3a598453ef3d57a6e861412f6616944e54: Status 404 returned error can't find the container with id 54317b9f414e40296f04cb452761cd3a598453ef3d57a6e861412f6616944e54 Jul 15 12:47:18.642167 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:18.642149 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:47:19.397645 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:19.397605 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d79rw/must-gather-5x8xm" event={"ID":"e9343e87-ff89-4e66-88a8-1bb0463dcb36","Type":"ContainerStarted","Data":"54317b9f414e40296f04cb452761cd3a598453ef3d57a6e861412f6616944e54"} Jul 15 12:47:20.406588 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:20.406550 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d79rw/must-gather-5x8xm" event={"ID":"e9343e87-ff89-4e66-88a8-1bb0463dcb36","Type":"ContainerStarted","Data":"e653bb6c41640c13895b7fdbe81b1bd89cc5bbb9e274d0d51ce0003c8aa95b19"} Jul 15 12:47:20.407042 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:20.406601 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d79rw/must-gather-5x8xm" event={"ID":"e9343e87-ff89-4e66-88a8-1bb0463dcb36","Type":"ContainerStarted","Data":"23cae855df4c50a95760dfa89835d1eb39fbab3f9b660192f4bcfaef4ae7452a"} Jul 15 12:47:20.423204 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:20.423153 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-d79rw/must-gather-5x8xm" podStartSLOduration=1.553901704 podStartE2EDuration="2.423133642s" totalImagesPullingTime="869.231938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:47:20.420340213 +0000 UTC m=+2907.833868564" watchObservedRunningTime="2026-07-15 12:47:20.423133642 +0000 UTC m=+2907.836661993" Jul 15 12:47:21.233846 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:21.233809 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ggb88_977f5c84-a6ea-4927-93ad-9dd0e23853a8/global-pull-secret-syncer/0.log" Jul 15 12:47:21.319091 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:21.319060 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fr6q8_88807a13-409e-4b95-b399-43ed30674850/konnectivity-agent/0.log" Jul 15 12:47:21.419543 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:21.419510 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-53.ec2.internal_72a235d1a8bfb42e6972f74d75275638/haproxy/0.log" Jul 15 12:47:26.994508 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:26.994296 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f894584f6-vnc5l_6d613f76-23d4-4392-a2e2-f8658440a072/authorino/0.log" Jul 15 12:47:27.115523 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:27.115484 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-zgfdl_d95a5f8a-9a7c-414b-a526-6db1b266c278/kuadrant-console-plugin/0.log" Jul 15 12:47:29.318014 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.317958 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-nwkhg_f9924fcc-57eb-421b-8255-4ee0fbb198cb/kube-state-metrics/0.log" Jul 15 12:47:29.341258 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.341230 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-nwkhg_f9924fcc-57eb-421b-8255-4ee0fbb198cb/kube-rbac-proxy-main/0.log" Jul 15 12:47:29.363017 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.362990 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-nwkhg_f9924fcc-57eb-421b-8255-4ee0fbb198cb/kube-rbac-proxy-self/0.log" Jul 15 12:47:29.389783 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.389663 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-794764c799-2ffql_028f65a3-a2d2-4e26-a8ae-a1a2fe5027f3/metrics-server/0.log" Jul 15 12:47:29.414416 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.414376 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-s5r7r_3a397af2-2ff1-4296-b8e9-3c7b3888aa81/monitoring-plugin/0.log" Jul 15 12:47:29.513501 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.513467 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chq6t_279e04c2-80d5-4f93-83b0-fb8ab35223f0/node-exporter/0.log" Jul 15 12:47:29.533202 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.533175 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chq6t_279e04c2-80d5-4f93-83b0-fb8ab35223f0/kube-rbac-proxy/0.log" Jul 15 12:47:29.561246 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.561219 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chq6t_279e04c2-80d5-4f93-83b0-fb8ab35223f0/init-textfile/0.log" Jul 15 12:47:29.660153 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.660124 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-qwrhj_54e8bb88-05f0-4569-968e-93f700263ce5/kube-rbac-proxy-main/0.log" Jul 15 12:47:29.680079 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.680042 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-qwrhj_54e8bb88-05f0-4569-968e-93f700263ce5/kube-rbac-proxy-self/0.log" Jul 15 12:47:29.699944 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.699917 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-qwrhj_54e8bb88-05f0-4569-968e-93f700263ce5/openshift-state-metrics/0.log" Jul 15 12:47:29.738359 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.738328 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_16cb1810-527b-4762-bb68-b35a23cbb19d/prometheus/0.log" Jul 15 12:47:29.759143 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.759119 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_16cb1810-527b-4762-bb68-b35a23cbb19d/config-reloader/0.log" Jul 15 12:47:29.786340 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.786310 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_16cb1810-527b-4762-bb68-b35a23cbb19d/thanos-sidecar/0.log" Jul 15 12:47:29.807513 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.807420 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_16cb1810-527b-4762-bb68-b35a23cbb19d/kube-rbac-proxy-web/0.log" Jul 15 12:47:29.827169 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.827135 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_16cb1810-527b-4762-bb68-b35a23cbb19d/kube-rbac-proxy/0.log" Jul 15 12:47:29.847195 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.847165 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_16cb1810-527b-4762-bb68-b35a23cbb19d/kube-rbac-proxy-thanos/0.log" Jul 15 12:47:29.877189 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:29.877126 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_16cb1810-527b-4762-bb68-b35a23cbb19d/init-config-reloader/0.log" Jul 15 12:47:30.057108 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.057037 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b49f799ff-mxqxz_8ce62b12-ec66-45e2-82b2-cf9910477b1c/thanos-query/0.log" Jul 15 12:47:30.076976 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.076945 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b49f799ff-mxqxz_8ce62b12-ec66-45e2-82b2-cf9910477b1c/kube-rbac-proxy-web/0.log" Jul 15 12:47:30.098330 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.098295 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b49f799ff-mxqxz_8ce62b12-ec66-45e2-82b2-cf9910477b1c/kube-rbac-proxy/0.log" Jul 15 12:47:30.117697 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.117665 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b49f799ff-mxqxz_8ce62b12-ec66-45e2-82b2-cf9910477b1c/prom-label-proxy/0.log" Jul 15 12:47:30.136889 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.136837 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b49f799ff-mxqxz_8ce62b12-ec66-45e2-82b2-cf9910477b1c/kube-rbac-proxy-rules/0.log" Jul 15 12:47:30.158224 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.158171 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b49f799ff-mxqxz_8ce62b12-ec66-45e2-82b2-cf9910477b1c/kube-rbac-proxy-metrics/0.log" Jul 15 12:47:30.859084 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.859049 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj"] Jul 15 12:47:30.864685 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.864652 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:30.892108 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.892070 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3881f5b-6a1e-42b1-a220-07d902ff5dc4-proc\") pod \"perf-node-gather-daemonset-v84lj\" (UID: \"f3881f5b-6a1e-42b1-a220-07d902ff5dc4\") " pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:30.892368 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.892346 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkd79\" (UniqueName: \"kubernetes.io/projected/f3881f5b-6a1e-42b1-a220-07d902ff5dc4-kube-api-access-bkd79\") pod \"perf-node-gather-daemonset-v84lj\" (UID: \"f3881f5b-6a1e-42b1-a220-07d902ff5dc4\") " pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:30.892518 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.892504 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3881f5b-6a1e-42b1-a220-07d902ff5dc4-podres\") pod \"perf-node-gather-daemonset-v84lj\" (UID: \"f3881f5b-6a1e-42b1-a220-07d902ff5dc4\") " pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:30.892701 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.892684 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3881f5b-6a1e-42b1-a220-07d902ff5dc4-sys\") pod \"perf-node-gather-daemonset-v84lj\" (UID: \"f3881f5b-6a1e-42b1-a220-07d902ff5dc4\") " pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:30.892841 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.892824 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3881f5b-6a1e-42b1-a220-07d902ff5dc4-lib-modules\") pod \"perf-node-gather-daemonset-v84lj\" (UID: \"f3881f5b-6a1e-42b1-a220-07d902ff5dc4\") " pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:30.894196 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.894163 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj"] Jul 15 12:47:30.993425 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.993383 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3881f5b-6a1e-42b1-a220-07d902ff5dc4-sys\") pod \"perf-node-gather-daemonset-v84lj\" (UID: \"f3881f5b-6a1e-42b1-a220-07d902ff5dc4\") " pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:30.993617 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.993433 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3881f5b-6a1e-42b1-a220-07d902ff5dc4-lib-modules\") pod \"perf-node-gather-daemonset-v84lj\" (UID: \"f3881f5b-6a1e-42b1-a220-07d902ff5dc4\") " pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:30.993617 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.993489 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3881f5b-6a1e-42b1-a220-07d902ff5dc4-proc\") pod \"perf-node-gather-daemonset-v84lj\" (UID: \"f3881f5b-6a1e-42b1-a220-07d902ff5dc4\") " pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:30.993617 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.993513 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkd79\" (UniqueName: \"kubernetes.io/projected/f3881f5b-6a1e-42b1-a220-07d902ff5dc4-kube-api-access-bkd79\") pod \"perf-node-gather-daemonset-v84lj\" (UID: \"f3881f5b-6a1e-42b1-a220-07d902ff5dc4\") " pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:30.993617 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.993533 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3881f5b-6a1e-42b1-a220-07d902ff5dc4-sys\") pod \"perf-node-gather-daemonset-v84lj\" (UID: \"f3881f5b-6a1e-42b1-a220-07d902ff5dc4\") " pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:30.993617 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.993595 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3881f5b-6a1e-42b1-a220-07d902ff5dc4-podres\") pod \"perf-node-gather-daemonset-v84lj\" (UID: \"f3881f5b-6a1e-42b1-a220-07d902ff5dc4\") " pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:30.993922 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.993656 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f3881f5b-6a1e-42b1-a220-07d902ff5dc4-proc\") pod \"perf-node-gather-daemonset-v84lj\" (UID: \"f3881f5b-6a1e-42b1-a220-07d902ff5dc4\") " pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:30.993922 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.993723 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3881f5b-6a1e-42b1-a220-07d902ff5dc4-lib-modules\") pod \"perf-node-gather-daemonset-v84lj\" (UID: \"f3881f5b-6a1e-42b1-a220-07d902ff5dc4\") " pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:30.994134 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:30.994113 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f3881f5b-6a1e-42b1-a220-07d902ff5dc4-podres\") pod \"perf-node-gather-daemonset-v84lj\" (UID: \"f3881f5b-6a1e-42b1-a220-07d902ff5dc4\") " pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:31.002501 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:31.002449 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkd79\" (UniqueName: \"kubernetes.io/projected/f3881f5b-6a1e-42b1-a220-07d902ff5dc4-kube-api-access-bkd79\") pod \"perf-node-gather-daemonset-v84lj\" (UID: \"f3881f5b-6a1e-42b1-a220-07d902ff5dc4\") " pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:31.181512 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:31.181412 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:31.397829 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:31.397730 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj"] Jul 15 12:47:31.401699 ip-10-0-131-53 kubenswrapper[2570]: W0715 12:47:31.401670 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf3881f5b_6a1e_42b1_a220_07d902ff5dc4.slice/crio-2313edca452b552249fdf7331c48ca972768171757d83bca882f120b7c6945e7 WatchSource:0}: Error finding container 2313edca452b552249fdf7331c48ca972768171757d83bca882f120b7c6945e7: Status 404 returned error can't find the container with id 2313edca452b552249fdf7331c48ca972768171757d83bca882f120b7c6945e7 Jul 15 12:47:31.489282 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:31.489133 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" event={"ID":"f3881f5b-6a1e-42b1-a220-07d902ff5dc4","Type":"ContainerStarted","Data":"2313edca452b552249fdf7331c48ca972768171757d83bca882f120b7c6945e7"} Jul 15 12:47:32.499319 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:32.499281 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" event={"ID":"f3881f5b-6a1e-42b1-a220-07d902ff5dc4","Type":"ContainerStarted","Data":"44fd01b1de39596b32cc66f94fe8426dcf3c33fb6b740f81a841cc2ad437c04a"} Jul 15 12:47:32.499828 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:32.499430 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:32.516851 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:32.516827 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-ff58cf98f-rklgw_d5e48be8-50e8-4b40-946e-60830b2369cd/console/0.log" Jul 15 12:47:32.518184 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:32.518143 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" podStartSLOduration=2.518128762 podStartE2EDuration="2.518128762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:47:32.515212738 +0000 UTC m=+2919.928741089" watchObservedRunningTime="2026-07-15 12:47:32.518128762 +0000 UTC m=+2919.931657112" Jul 15 12:47:34.216539 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:34.216509 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-24s4z_a92dead1-e314-46f5-9a12-c34efd573904/dns/0.log" Jul 15 12:47:34.245146 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:34.245111 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-24s4z_a92dead1-e314-46f5-9a12-c34efd573904/kube-rbac-proxy/0.log" Jul 15 12:47:34.388388 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:34.388344 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9zcb8_3a23f0bc-7022-4e72-b3cf-0a63b6e681f8/dns-node-resolver/0.log" Jul 15 12:47:35.048815 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:35.048786 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-snrv8_e507f9d6-cd28-4d22-9dfe-5869963a8c48/node-ca/0.log" Jul 15 12:47:36.258529 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:36.258496 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d7f886b65-v8fmz_621592bf-b388-465a-843a-2c803e0c6c1c/kube-auth-proxy/0.log" Jul 15 12:47:36.438631 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:36.438602 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c864b74c8-z6ccx_f8699bdf-f732-4921-a592-483bea1c84fb/router/0.log" Jul 15 12:47:37.255451 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:37.255418 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5zvqc_8b959ddc-0f8a-473d-a311-afcfa283fed3/serve-healthcheck-canary/0.log" Jul 15 12:47:37.807403 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:37.807372 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-289vh_144f5339-e458-46aa-acbe-f43a1e4dbec9/kube-rbac-proxy/0.log" Jul 15 12:47:37.829917 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:37.829867 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-289vh_144f5339-e458-46aa-acbe-f43a1e4dbec9/exporter/0.log" Jul 15 12:47:37.850648 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:37.850617 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-289vh_144f5339-e458-46aa-acbe-f43a1e4dbec9/extractor/0.log" Jul 15 12:47:38.519433 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:38.519407 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-d79rw/perf-node-gather-daemonset-v84lj" Jul 15 12:47:40.650530 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:40.650496 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55594846d9-q2qsp_4adc5ad8-bd3d-420d-9e65-a7b7d4b47f02/manager/0.log" Jul 15 12:47:40.806304 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:40.806264 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694c4fcbc8-6vv9f_69ea5e3a-62a0-476b-80de-32af2b08ff93/manager/0.log" Jul 15 12:47:40.826063 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:40.826031 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694c4fcbc8-6vv9f_69ea5e3a-62a0-476b-80de-32af2b08ff93/copy-manifests/0.log" Jul 15 12:47:42.015327 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:42.015286 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d7f886b65-v8fmz_621592bf-b388-465a-843a-2c803e0c6c1c/kube-auth-proxy/0.log" Jul 15 12:47:42.163920 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:42.163886 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c864b74c8-z6ccx_f8699bdf-f732-4921-a592-483bea1c84fb/router/0.log" Jul 15 12:47:42.237215 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:42.237169 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5dc89645fd-k9j5c_cc7e65d9-a799-4475-b9b4-450d0c2627ce/manager/0.log" Jul 15 12:47:43.547961 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:43.547883 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d7f886b65-v8fmz_621592bf-b388-465a-843a-2c803e0c6c1c/kube-auth-proxy/0.log" Jul 15 12:47:43.694028 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:43.693988 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c864b74c8-z6ccx_f8699bdf-f732-4921-a592-483bea1c84fb/router/0.log" Jul 15 12:47:47.801650 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:47.801614 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-ftmvb_74bcecd1-b162-4ce3-b731-d47b223860a8/kube-storage-version-migrator-operator/1.log" Jul 15 12:47:47.802786 ip-10-0-131-53 kubenswrapper[2570]: I0715 12:47:47.802764 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-ftmvb_74bcecd1-b162-4ce3-b731-d47b223860a8/kube-storage-version-migrator-operator/0.log"